[==========] Running 57 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 31 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:03:33 dist-test-slave-8n77 krb5kdc[2846](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:03:33 dist-test-slave-8n77 krb5kdc[2846](info): set up 2 sockets
Apr 29 14:03:33 dist-test-slave-8n77 krb5kdc[2846](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260429 14:03:35.548207 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:03:35 dist-test-slave-8n77 krb5kdc[2846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:03:35Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:03:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:03:35.710642 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:38439
--webserver_interface=127.2.197.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:40775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:38439
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:03:35.793911 2862 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:03:35.794173 2862 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:03:35.794211 2862 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:03:35.795845 2862 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:03:35.795933 2862 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:03:35.795955 2862 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:03:35.795977 2862 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:03:35.796010 2862 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:03:35.797752 2862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:40775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:38439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:38439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:03:35.798044 2862 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:03:35.798245 2862 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:03:35.801349 2868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:03:35.801282 2862 server_base.cc:1061] running on GCE node
W20260429 14:03:35.801282 2870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:03:35.801301 2867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:03:35.801843 2862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:03:35.802103 2862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:03:35.803288 2862 hybrid_clock.cc:648] HybridClock initialized: now 1777471415803277 us; error 30 us; skew 500 ppm
Apr 29 14:03:35 dist-test-slave-8n77 krb5kdc[2846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471415, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:03:35.805932 2862 init.cc:377] Logged in from keytab as kudu/127.2.197.254@KRBTEST.COM (short username kudu)
I20260429 14:03:35.806577 2862 webserver.cc:492] Webserver started at http://127.2.197.254:34361/ using document root <none> and password file <none>
I20260429 14:03:35.806782 2862 fs_manager.cc:362] Metadata directory not provided
I20260429 14:03:35.806845 2862 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:03:35.806962 2862 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:03:35.807938 2862 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "435a6ee6ecfd4af2afea2d4fd739b39f"
format_stamp: "Formatted at 2026-04-29 14:03:35 on dist-test-slave-8n77"
server_key: "838741ff2ac0e4e8c6b12eb47db22d61"
server_key_iv: "6ba515e5462c8c5049392973f057998d"
server_key_version: "encryptionkey@0"
I20260429 14:03:35.808282 2862 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "435a6ee6ecfd4af2afea2d4fd739b39f"
format_stamp: "Formatted at 2026-04-29 14:03:35 on dist-test-slave-8n77"
server_key: "838741ff2ac0e4e8c6b12eb47db22d61"
server_key_iv: "6ba515e5462c8c5049392973f057998d"
server_key_version: "encryptionkey@0"
I20260429 14:03:35.809582 2862 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260429 14:03:35.810444 2877 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:35.810683 2862 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:03:35.810770 2862 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "435a6ee6ecfd4af2afea2d4fd739b39f"
format_stamp: "Formatted at 2026-04-29 14:03:35 on dist-test-slave-8n77"
server_key: "838741ff2ac0e4e8c6b12eb47db22d61"
server_key_iv: "6ba515e5462c8c5049392973f057998d"
server_key_version: "encryptionkey@0"
I20260429 14:03:35.810858 2862 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:03:35.822216 2862 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:03:35.828110 2862 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:03:35.828301 2862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:03:35.832698 2862 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:38439
I20260429 14:03:35.832774 2929 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:38439 every 8 connection(s)
I20260429 14:03:35.833133 2862 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:03:35.833762 2930 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:03:35.835608 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 2862
I20260429 14:03:35.835777 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:03:35.836354 2839 external_mini_cluster.cc:1468] Setting key a9ad6bd500eacec2ec9b049e5798074b
I20260429 14:03:35.836354 2930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 435a6ee6ecfd4af2afea2d4fd739b39f: Bootstrap starting.
I20260429 14:03:35.837095 2930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 435a6ee6ecfd4af2afea2d4fd739b39f: Neither blocks nor log segments found. Creating new log.
I20260429 14:03:35.837457 2930 log.cc:826] T 00000000000000000000000000000000 P 435a6ee6ecfd4af2afea2d4fd739b39f: Log is configured to *not* fsync() on all Append() calls
I20260429 14:03:35.838167 2930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 435a6ee6ecfd4af2afea2d4fd739b39f: No bootstrap required, opened a new log
I20260429 14:03:35.839455 2930 raft_consensus.cc:359] T 00000000000000000000000000000000 P 435a6ee6ecfd4af2afea2d4fd739b39f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "435a6ee6ecfd4af2afea2d4fd739b39f" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 38439 } }
I20260429 14:03:35.839604 2930 raft_consensus.cc:385] T 00000000000000000000000000000000 P 435a6ee6ecfd4af2afea2d4fd739b39f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:03:35.839646 2930 raft_consensus.cc:740] T 00000000000000000000000000000000 P 435a6ee6ecfd4af2afea2d4fd739b39f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 435a6ee6ecfd4af2afea2d4fd739b39f, State: Initialized, Role: FOLLOWER
I20260429 14:03:35.839779 2930 consensus_queue.cc:260] T 00000000000000000000000000000000 P 435a6ee6ecfd4af2afea2d4fd739b39f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "435a6ee6ecfd4af2afea2d4fd739b39f" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 38439 } }
I20260429 14:03:35.839862 2930 raft_consensus.cc:399] T 00000000000000000000000000000000 P 435a6ee6ecfd4af2afea2d4fd739b39f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:03:35.839906 2930 raft_consensus.cc:493] T 00000000000000000000000000000000 P 435a6ee6ecfd4af2afea2d4fd739b39f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:03:35.839975 2930 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 435a6ee6ecfd4af2afea2d4fd739b39f [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:03:35.840550 2930 raft_consensus.cc:515] T 00000000000000000000000000000000 P 435a6ee6ecfd4af2afea2d4fd739b39f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "435a6ee6ecfd4af2afea2d4fd739b39f" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 38439 } }
I20260429 14:03:35.840726 2930 leader_election.cc:304] T 00000000000000000000000000000000 P 435a6ee6ecfd4af2afea2d4fd739b39f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 435a6ee6ecfd4af2afea2d4fd739b39f; no voters:
I20260429 14:03:35.840901 2930 leader_election.cc:290] T 00000000000000000000000000000000 P 435a6ee6ecfd4af2afea2d4fd739b39f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:03:35.840967 2935 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 435a6ee6ecfd4af2afea2d4fd739b39f [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:03:35.841189 2935 raft_consensus.cc:697] T 00000000000000000000000000000000 P 435a6ee6ecfd4af2afea2d4fd739b39f [term 1 LEADER]: Becoming Leader. State: Replica: 435a6ee6ecfd4af2afea2d4fd739b39f, State: Running, Role: LEADER
I20260429 14:03:35.841197 2930 sys_catalog.cc:565] T 00000000000000000000000000000000 P 435a6ee6ecfd4af2afea2d4fd739b39f [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:03:35.841394 2935 consensus_queue.cc:237] T 00000000000000000000000000000000 P 435a6ee6ecfd4af2afea2d4fd739b39f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "435a6ee6ecfd4af2afea2d4fd739b39f" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 38439 } }
I20260429 14:03:35.841850 2935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 435a6ee6ecfd4af2afea2d4fd739b39f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 435a6ee6ecfd4af2afea2d4fd739b39f. Latest consensus state: current_term: 1 leader_uuid: "435a6ee6ecfd4af2afea2d4fd739b39f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "435a6ee6ecfd4af2afea2d4fd739b39f" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 38439 } } }
I20260429 14:03:35.841958 2935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 435a6ee6ecfd4af2afea2d4fd739b39f [sys.catalog]: This master's current role is: LEADER
I20260429 14:03:35.842242 2944 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:03:35.843189 2937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 435a6ee6ecfd4af2afea2d4fd739b39f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "435a6ee6ecfd4af2afea2d4fd739b39f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "435a6ee6ecfd4af2afea2d4fd739b39f" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 38439 } } }
I20260429 14:03:35.843356 2937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 435a6ee6ecfd4af2afea2d4fd739b39f [sys.catalog]: This master's current role is: LEADER
I20260429 14:03:35.844005 2944 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:03:35.846233 2944 catalog_manager.cc:1357] Generated new cluster ID: 0bcbdc3fd759410796a0decf7b589a4e
I20260429 14:03:35.846305 2944 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Apr 29 14:03:35 dist-test-slave-8n77 krb5kdc[2846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:03:35.852039 2933 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:35.837197 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51612 (local address 127.2.197.254:38439)
0429 14:03:35.837419 (+ 222us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:35.837430 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:35.837588 (+ 158us) server_negotiation.cc:408] Connection header received
0429 14:03:35.839459 (+ 1871us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:35.839467 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:35.839538 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:35.839706 (+ 168us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:35.841578 (+ 1872us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:35.842805 (+ 1227us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:35.844488 (+ 1683us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:35.844739 (+ 251us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:35.847835 (+ 3096us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:35.847842 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:35.847842 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:35.847856 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:35.850194 (+ 2338us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:35.850733 (+ 539us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:35.850747 (+ 14us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:35.850747 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:35.850810 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:35.851013 (+ 203us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:35.851014 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:35.851014 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:35.851217 (+ 203us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:35.851326 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:35.851704 (+ 378us) server_negotiation.cc:300] Negotiation successful
0429 14:03:35.851859 (+ 155us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":125,"threads_started":1}
I20260429 14:03:35.856081 2944 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:03:35.856832 2944 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:03:35.864137 2944 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 435a6ee6ecfd4af2afea2d4fd739b39f: Generated new TSK 0
I20260429 14:03:35.864377 2944 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:03:35.930089 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.193:0
--local_ip_for_outbound_sockets=127.2.197.193
--webserver_interface=127.2.197.193
--webserver_port=0
--tserver_master_addrs=127.2.197.254:38439
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:40775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:03:36.012905 2958 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:03:36.013144 2958 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:03:36.013187 2958 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:03:36.015027 2958 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:03:36.015105 2958 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:03:36.015153 2958 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.193
I20260429 14:03:36.016949 2958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:40775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.197.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:38439
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.193
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:03:36.017223 2958 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:03:36.017448 2958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:03:36.020591 2964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:03:36.020645 2966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:03:36.020601 2963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:03:36.021064 2958 server_base.cc:1061] running on GCE node
I20260429 14:03:36.021314 2958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:03:36.021634 2958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:03:36.022893 2958 hybrid_clock.cc:648] HybridClock initialized: now 1777471416022863 us; error 47 us; skew 500 ppm
Apr 29 14:03:36 dist-test-slave-8n77 krb5kdc[2846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471416, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:03:36.025594 2958 init.cc:377] Logged in from keytab as kudu/127.2.197.193@KRBTEST.COM (short username kudu)
I20260429 14:03:36.026177 2958 webserver.cc:492] Webserver started at http://127.2.197.193:37123/ using document root <none> and password file <none>
I20260429 14:03:36.026417 2958 fs_manager.cc:362] Metadata directory not provided
I20260429 14:03:36.026461 2958 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:03:36.026564 2958 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:03:36.027555 2958 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-0/data/instance:
uuid: "0d5bae85f7eb45d5825f0c1f6f25be4c"
format_stamp: "Formatted at 2026-04-29 14:03:36 on dist-test-slave-8n77"
server_key: "55ddc90fad12ecfa62a9c41787b050a2"
server_key_iv: "63320caab3e564e76fafcf8c6089179e"
server_key_version: "encryptionkey@0"
I20260429 14:03:36.027899 2958 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance:
uuid: "0d5bae85f7eb45d5825f0c1f6f25be4c"
format_stamp: "Formatted at 2026-04-29 14:03:36 on dist-test-slave-8n77"
server_key: "55ddc90fad12ecfa62a9c41787b050a2"
server_key_iv: "63320caab3e564e76fafcf8c6089179e"
server_key_version: "encryptionkey@0"
I20260429 14:03:36.029349 2958 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260429 14:03:36.030422 2973 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:36.030777 2958 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:03:36.030916 2958 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-0/wal
uuid: "0d5bae85f7eb45d5825f0c1f6f25be4c"
format_stamp: "Formatted at 2026-04-29 14:03:36 on dist-test-slave-8n77"
server_key: "55ddc90fad12ecfa62a9c41787b050a2"
server_key_iv: "63320caab3e564e76fafcf8c6089179e"
server_key_version: "encryptionkey@0"
I20260429 14:03:36.031005 2958 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:03:36.038041 2958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:03:36.040246 2958 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:03:36.040407 2958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:03:36.040643 2958 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:03:36.040959 2958 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:03:36.041015 2958 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:36.041077 2958 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:03:36.041110 2958 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:36.047523 2958 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.193:34361
I20260429 14:03:36.047573 3086 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.193:34361 every 8 connection(s)
I20260429 14:03:36.047876 2958 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
Apr 29 14:03:36 dist-test-slave-8n77 krb5kdc[2846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471416, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:03:36.055421 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 2958
I20260429 14:03:36.055546 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance
I20260429 14:03:36.055724 2839 external_mini_cluster.cc:1468] Setting key 7ff7e3258738c6d04883ee3dad9a7a88
I20260429 14:03:36.057025 2933 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:36.048632 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:40863 (local address 127.2.197.254:38439)
0429 14:03:36.048732 (+ 100us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:36.048735 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:36.048901 (+ 166us) server_negotiation.cc:408] Connection header received
0429 14:03:36.049418 (+ 517us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:36.049419 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:36.049435 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:36.049501 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:36.050247 (+ 746us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:36.050836 (+ 589us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:36.051414 (+ 578us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:36.051599 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:36.053920 (+ 2321us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:36.053925 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:36.053925 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:36.053934 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:36.055494 (+ 1560us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:36.056027 (+ 533us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:36.056028 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:36.056028 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:36.056052 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:36.056262 (+ 210us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:36.056263 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:36.056263 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:36.056373 (+ 110us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:36.056437 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:36.056800 (+ 363us) server_negotiation.cc:300] Negotiation successful
0429 14:03:36.056932 (+ 132us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260429 14:03:36.057147 3089 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:36.048575 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:38439 (local address 127.2.197.193:40863)
0429 14:03:36.048757 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:36.048796 (+ 39us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:36.049322 (+ 526us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:36.049567 (+ 245us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:36.049571 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:36.049882 (+ 311us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:03:36.050148 (+ 266us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:36.050150 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:36.050944 (+ 794us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:36.050945 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:36.051329 (+ 384us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:36.051331 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:36.051460 (+ 129us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:36.052412 (+ 952us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:03:36.052421 (+ 9us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:03:36.053826 (+ 1405us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:03:36.055642 (+ 1816us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:36.055649 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:36.055652 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:36.055933 (+ 281us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:36.056103 (+ 170us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:36.056104 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:36.056105 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:36.056184 (+ 79us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:36.056472 (+ 288us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:03:36.056478 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:03:36.056642 (+ 164us) client_negotiation.cc:770] Sending connection context
0429 14:03:36.056767 (+ 125us) client_negotiation.cc:241] Negotiation successful
0429 14:03:36.056928 (+ 161us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":95,"threads_started":1}
I20260429 14:03:36.057626 3087 heartbeater.cc:344] Connected to a master server at 127.2.197.254:38439
I20260429 14:03:36.057716 3087 heartbeater.cc:461] Registering TS with master...
I20260429 14:03:36.057969 3087 heartbeater.cc:507] Master 127.2.197.254:38439 requested a full tablet report, sending...
I20260429 14:03:36.058534 2894 ts_manager.cc:194] Registered new tserver with Master: 0d5bae85f7eb45d5825f0c1f6f25be4c (127.2.197.193:34361)
I20260429 14:03:36.059231 2894 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.193@KRBTEST.COM'} at 127.2.197.193:40863
WARNING: no policy specified for kudu/127.2.197.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:03:36.113351 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.194:0
--local_ip_for_outbound_sockets=127.2.197.194
--webserver_interface=127.2.197.194
--webserver_port=0
--tserver_master_addrs=127.2.197.254:38439
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:40775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:03:36.191988 3094 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:03:36.192193 3094 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:03:36.192220 3094 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:03:36.193863 3094 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:03:36.193920 3094 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:03:36.193950 3094 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.194
I20260429 14:03:36.195765 3094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:40775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.197.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:38439
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.194
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:03:36.196022 3094 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:03:36.196228 3094 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:03:36.199203 3099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:03:36.199221 3100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:03:36.199277 3094 server_base.cc:1061] running on GCE node
W20260429 14:03:36.199203 3102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:03:36.199685 3094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:03:36.199952 3094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:03:36.201128 3094 hybrid_clock.cc:648] HybridClock initialized: now 1777471416201092 us; error 50 us; skew 500 ppm
Apr 29 14:03:36 dist-test-slave-8n77 krb5kdc[2846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471416, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:03:36.203745 3094 init.cc:377] Logged in from keytab as kudu/127.2.197.194@KRBTEST.COM (short username kudu)
I20260429 14:03:36.204325 3094 webserver.cc:492] Webserver started at http://127.2.197.194:45267/ using document root <none> and password file <none>
I20260429 14:03:36.204555 3094 fs_manager.cc:362] Metadata directory not provided
I20260429 14:03:36.204625 3094 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:03:36.204770 3094 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:03:36.205808 3094 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-1/data/instance:
uuid: "69267b7de8b7479387f636d2a3d37831"
format_stamp: "Formatted at 2026-04-29 14:03:36 on dist-test-slave-8n77"
server_key: "9bf5907cac030cf7045de17d803bcbe3"
server_key_iv: "22a12654a4a4801fd4173a59b63ca7de"
server_key_version: "encryptionkey@0"
I20260429 14:03:36.206159 3094 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance:
uuid: "69267b7de8b7479387f636d2a3d37831"
format_stamp: "Formatted at 2026-04-29 14:03:36 on dist-test-slave-8n77"
server_key: "9bf5907cac030cf7045de17d803bcbe3"
server_key_iv: "22a12654a4a4801fd4173a59b63ca7de"
server_key_version: "encryptionkey@0"
I20260429 14:03:36.207702 3094 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:03:36.208583 3109 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:36.208823 3094 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:03:36.208913 3094 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-1/wal
uuid: "69267b7de8b7479387f636d2a3d37831"
format_stamp: "Formatted at 2026-04-29 14:03:36 on dist-test-slave-8n77"
server_key: "9bf5907cac030cf7045de17d803bcbe3"
server_key_iv: "22a12654a4a4801fd4173a59b63ca7de"
server_key_version: "encryptionkey@0"
I20260429 14:03:36.209002 3094 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:03:36.235917 3094 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:03:36.238183 3094 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:03:36.238579 3094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:03:36.238855 3094 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:03:36.239194 3094 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:03:36.239229 3094 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:36.239252 3094 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:03:36.239265 3094 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:36.246325 3094 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.194:40235
I20260429 14:03:36.246407 3222 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.194:40235 every 8 connection(s)
I20260429 14:03:36.246829 3094 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
I20260429 14:03:36.248804 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 3094
I20260429 14:03:36.248912 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance
I20260429 14:03:36.249082 2839 external_mini_cluster.cc:1468] Setting key b1dfba56862926dd2e77cb57aa11e1c9
Apr 29 14:03:36 dist-test-slave-8n77 krb5kdc[2846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471416, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:03:36.257421 3225 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:36.247650 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:38439 (local address 127.2.197.194:35787)
0429 14:03:36.247905 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:36.247944 (+ 39us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:36.248401 (+ 457us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:36.248648 (+ 247us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:36.248652 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:36.249030 (+ 378us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:03:36.249388 (+ 358us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:36.249392 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:36.250416 (+ 1024us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:36.250417 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:36.250799 (+ 382us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:36.250801 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:36.250899 (+ 98us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:36.251862 (+ 963us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:03:36.251872 (+ 10us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:03:36.253823 (+ 1951us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:03:36.256000 (+ 2177us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:36.256005 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:36.256008 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:36.256228 (+ 220us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:36.256435 (+ 207us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:36.256436 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:36.256436 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:36.256601 (+ 165us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:36.256938 (+ 337us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:03:36.256942 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:03:36.257053 (+ 111us) client_negotiation.cc:770] Sending connection context
0429 14:03:36.257148 (+ 95us) client_negotiation.cc:241] Negotiation successful
0429 14:03:36.257263 (+ 115us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":96,"threads_started":1}
I20260429 14:03:36.257422 2933 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:36.247796 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:35787 (local address 127.2.197.254:38439)
0429 14:03:36.247886 (+ 90us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:36.247889 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:36.248037 (+ 148us) server_negotiation.cc:408] Connection header received
0429 14:03:36.248491 (+ 454us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:36.248492 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:36.248503 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:36.248569 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:36.249478 (+ 909us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:36.250323 (+ 845us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:36.250895 (+ 572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:36.251120 (+ 225us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:36.253971 (+ 2851us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:36.253978 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:36.253978 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:36.253990 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:36.255888 (+ 1898us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:36.256323 (+ 435us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:36.256325 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:36.256325 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:36.256358 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:36.256708 (+ 350us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:36.256709 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:36.256710 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:36.256857 (+ 147us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:36.256947 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:36.257202 (+ 255us) server_negotiation.cc:300] Negotiation successful
0429 14:03:36.257308 (+ 106us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260429 14:03:36.258036 3223 heartbeater.cc:344] Connected to a master server at 127.2.197.254:38439
I20260429 14:03:36.258224 3223 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for kudu/127.2.197.195@KRBTEST.COM; defaulting to no policy
I20260429 14:03:36.258545 3223 heartbeater.cc:507] Master 127.2.197.254:38439 requested a full tablet report, sending...
I20260429 14:03:36.259012 2894 ts_manager.cc:194] Registered new tserver with Master: 69267b7de8b7479387f636d2a3d37831 (127.2.197.194:40235)
I20260429 14:03:36.259445 2894 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.194@KRBTEST.COM'} at 127.2.197.194:35787
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:03:36.302551 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.195:0
--local_ip_for_outbound_sockets=127.2.197.195
--webserver_interface=127.2.197.195
--webserver_port=0
--tserver_master_addrs=127.2.197.254:38439
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:40775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:03:36.382959 3230 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:03:36.383162 3230 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:03:36.383188 3230 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:03:36.384801 3230 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:03:36.384853 3230 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:03:36.384882 3230 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.195
I20260429 14:03:36.386641 3230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:40775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.197.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:38439
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.195
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:03:36.386951 3230 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:03:36.387172 3230 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:03:36.389816 3238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:03:36.389814 3235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:03:36.389810 3236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:03:36.389971 3230 server_base.cc:1061] running on GCE node
I20260429 14:03:36.390233 3230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:03:36.390514 3230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:03:36.391696 3230 hybrid_clock.cc:648] HybridClock initialized: now 1777471416391652 us; error 54 us; skew 500 ppm
Apr 29 14:03:36 dist-test-slave-8n77 krb5kdc[2846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471416, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:03:36.394536 3230 init.cc:377] Logged in from keytab as kudu/127.2.197.195@KRBTEST.COM (short username kudu)
I20260429 14:03:36.395146 3230 webserver.cc:492] Webserver started at http://127.2.197.195:41045/ using document root <none> and password file <none>
I20260429 14:03:36.395356 3230 fs_manager.cc:362] Metadata directory not provided
I20260429 14:03:36.395419 3230 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:03:36.395530 3230 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:03:36.396514 3230 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-2/data/instance:
uuid: "dbd8729f9624470fb1d1bb973dfe793a"
format_stamp: "Formatted at 2026-04-29 14:03:36 on dist-test-slave-8n77"
server_key: "30c5cb1b503f5c94894f605397d6125f"
server_key_iv: "1ae797705308762427819d08f1bd944b"
server_key_version: "encryptionkey@0"
I20260429 14:03:36.396874 3230 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance:
uuid: "dbd8729f9624470fb1d1bb973dfe793a"
format_stamp: "Formatted at 2026-04-29 14:03:36 on dist-test-slave-8n77"
server_key: "30c5cb1b503f5c94894f605397d6125f"
server_key_iv: "1ae797705308762427819d08f1bd944b"
server_key_version: "encryptionkey@0"
I20260429 14:03:36.398198 3230 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260429 14:03:36.399075 3245 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:36.399253 3230 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:03:36.399338 3230 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-2/wal
uuid: "dbd8729f9624470fb1d1bb973dfe793a"
format_stamp: "Formatted at 2026-04-29 14:03:36 on dist-test-slave-8n77"
server_key: "30c5cb1b503f5c94894f605397d6125f"
server_key_iv: "1ae797705308762427819d08f1bd944b"
server_key_version: "encryptionkey@0"
I20260429 14:03:36.399420 3230 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:03:36.421907 3230 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:03:36.424129 3230 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:03:36.424311 3230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:03:36.424582 3230 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:03:36.424906 3230 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:03:36.424961 3230 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:36.424997 3230 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:03:36.425045 3230 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:36.431191 3230 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.195:38961
I20260429 14:03:36.431231 3358 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.195:38961 every 8 connection(s)
I20260429 14:03:36.431536 3230 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
Apr 29 14:03:36 dist-test-slave-8n77 krb5kdc[2846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471416, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:03:36.437335 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 3230
I20260429 14:03:36.437489 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance
I20260429 14:03:36.437659 2839 external_mini_cluster.cc:1468] Setting key 1aefe1317a1576bea3654a79bdfc3875
I20260429 14:03:36.441032 2933 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:36.432283 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:45647 (local address 127.2.197.254:38439)
0429 14:03:36.432346 (+ 63us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:36.432349 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:36.432541 (+ 192us) server_negotiation.cc:408] Connection header received
0429 14:03:36.433055 (+ 514us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:36.433057 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:36.433069 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:36.433132 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:36.433855 (+ 723us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:36.434489 (+ 634us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:36.435044 (+ 555us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:36.435211 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:36.438303 (+ 3092us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:36.438307 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:36.438308 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:36.438317 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:36.439900 (+ 1583us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:36.440263 (+ 363us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:36.440264 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:36.440265 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:36.440285 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:36.440443 (+ 158us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:36.440444 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:36.440444 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:36.440538 (+ 94us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:36.440589 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:36.440881 (+ 292us) server_negotiation.cc:300] Negotiation successful
0429 14:03:36.440966 (+ 85us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260429 14:03:36.441193 3361 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:36.432279 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:38439 (local address 127.2.197.195:45647)
0429 14:03:36.432428 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:36.432461 (+ 33us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:36.432960 (+ 499us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:36.433197 (+ 237us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:36.433201 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:36.433495 (+ 294us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:03:36.433747 (+ 252us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:36.433750 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:36.434583 (+ 833us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:36.434585 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:36.434963 (+ 378us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:36.434965 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:36.435093 (+ 128us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:36.436036 (+ 943us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:03:36.436045 (+ 9us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:03:36.438166 (+ 2121us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:03:36.439981 (+ 1815us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:36.439986 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:36.439988 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:36.440195 (+ 207us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:36.440342 (+ 147us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:36.440342 (+ 0us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:36.440343 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:36.440396 (+ 53us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:36.440620 (+ 224us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:03:36.440625 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:03:36.440751 (+ 126us) client_negotiation.cc:770] Sending connection context
0429 14:03:36.440854 (+ 103us) client_negotiation.cc:241] Negotiation successful
0429 14:03:36.440979 (+ 125us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":75,"threads_started":1}
I20260429 14:03:36.441781 3359 heartbeater.cc:344] Connected to a master server at 127.2.197.254:38439
I20260429 14:03:36.441896 3359 heartbeater.cc:461] Registering TS with master...
I20260429 14:03:36.442055 3359 heartbeater.cc:507] Master 127.2.197.254:38439 requested a full tablet report, sending...
I20260429 14:03:36.442567 2894 ts_manager.cc:194] Registered new tserver with Master: dbd8729f9624470fb1d1bb973dfe793a (127.2.197.195:38961)
I20260429 14:03:36.442929 2894 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.195@KRBTEST.COM'} at 127.2.197.195:45647
I20260429 14:03:36.450445 2839 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Apr 29 14:03:36 dist-test-slave-8n77 krb5kdc[2846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471416, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Apr 29 14:03:36 dist-test-slave-8n77 krb5kdc[2846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471416, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.197.193@KRBTEST.COM
I20260429 14:03:36.474591 3366 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:36.465095 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57872 (local address 127.2.197.193:34361)
0429 14:03:36.465374 (+ 279us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:36.465385 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:36.465591 (+ 206us) server_negotiation.cc:408] Connection header received
0429 14:03:36.465697 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:36.465702 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:36.465722 (+ 20us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:36.465813 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:36.466861 (+ 1048us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:36.467639 (+ 778us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:36.468251 (+ 612us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:36.468509 (+ 258us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:36.470657 (+ 2148us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:36.470694 (+ 37us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:36.470694 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:36.470701 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:36.472891 (+ 2190us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:36.473344 (+ 453us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:36.473352 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:36.473352 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:36.473530 (+ 178us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:36.473848 (+ 318us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:36.473849 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:36.473850 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:36.473991 (+ 141us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:36.474156 (+ 165us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:36.474321 (+ 165us) server_negotiation.cc:300] Negotiation successful
0429 14:03:36.474445 (+ 124us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":79,"threads_started":1}
W20260429 14:03:36.475113 3001 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57872
Apr 29 14:03:36 dist-test-slave-8n77 krb5kdc[2846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Apr 29 14:03:36 dist-test-slave-8n77 krb5kdc[2846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.193@KRBTEST.COM
I20260429 14:03:36.498618 3366 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:36.491009 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57874 (local address 127.2.197.193:34361)
0429 14:03:36.491096 (+ 87us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:36.491098 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:36.491114 (+ 16us) server_negotiation.cc:408] Connection header received
0429 14:03:36.491208 (+ 94us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:36.491209 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:36.491223 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:36.491290 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:36.492067 (+ 777us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:36.492622 (+ 555us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:36.493246 (+ 624us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:36.493437 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:36.495825 (+ 2388us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:36.495828 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:36.495829 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:36.495836 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:36.497344 (+ 1508us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:36.497764 (+ 420us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:36.497765 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:36.497766 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:36.497787 (+ 21us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:36.498052 (+ 265us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:36.498053 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:36.498053 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:36.498152 (+ 99us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:36.498241 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:36.498454 (+ 213us) server_negotiation.cc:300] Negotiation successful
0429 14:03:36.498547 (+ 93us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260429 14:03:36.499374 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 2958
I20260429 14:03:36.504887 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 3094
I20260429 14:03:36.511327 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 3230
I20260429 14:03:36.516425 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 2862
2026-04-29T14:03:36Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3026 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:03:36 dist-test-slave-8n77 krb5kdc[3373](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:03:36 dist-test-slave-8n77 krb5kdc[3373](info): set up 2 sockets
Apr 29 14:03:36 dist-test-slave-8n77 krb5kdc[3373](info): commencing operation
krb5kdc: starting...
W20260429 14:03:38.555609 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:03:38 dist-test-slave-8n77 krb5kdc[3373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:03:38Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:03:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:03:38.706682 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:42069
--webserver_interface=127.2.197.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:33077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:42069
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:03:38.789681 3389 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:03:38.789940 3389 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:03:38.789981 3389 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:03:38.791680 3389 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:03:38.791772 3389 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:03:38.791796 3389 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:03:38.791819 3389 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:03:38.791846 3389 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:03:38.793628 3389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:33077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:42069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:42069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:03:38.793932 3389 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:03:38.794128 3389 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:03:38.797153 3395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:03:38.797158 3397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:03:38.797154 3394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:03:38.797904 3389 server_base.cc:1061] running on GCE node
I20260429 14:03:38.798372 3389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:03:38.798671 3389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:03:38.799850 3389 hybrid_clock.cc:648] HybridClock initialized: now 1777471418799825 us; error 42 us; skew 500 ppm
Apr 29 14:03:38 dist-test-slave-8n77 krb5kdc[3373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471418, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:03:38.803017 3389 init.cc:377] Logged in from keytab as kudu/127.2.197.254@KRBTEST.COM (short username kudu)
I20260429 14:03:38.803785 3389 webserver.cc:492] Webserver started at http://127.2.197.254:34781/ using document root <none> and password file <none>
I20260429 14:03:38.803964 3389 fs_manager.cc:362] Metadata directory not provided
I20260429 14:03:38.804006 3389 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:03:38.804090 3389 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:03:38.805850 3389 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "8b0390cda90247b3a4bb9e3976aebcd2"
format_stamp: "Formatted at 2026-04-29 14:03:38 on dist-test-slave-8n77"
server_key: "a90227d39f1d6dcf8d7fee7a38426d4f"
server_key_iv: "11056c5ea4a2278d0210e24b1592ea01"
server_key_version: "encryptionkey@0"
I20260429 14:03:38.806422 3389 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "8b0390cda90247b3a4bb9e3976aebcd2"
format_stamp: "Formatted at 2026-04-29 14:03:38 on dist-test-slave-8n77"
server_key: "a90227d39f1d6dcf8d7fee7a38426d4f"
server_key_iv: "11056c5ea4a2278d0210e24b1592ea01"
server_key_version: "encryptionkey@0"
I20260429 14:03:38.808357 3389 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260429 14:03:38.809384 3404 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:38.809760 3389 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:03:38.809885 3389 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "8b0390cda90247b3a4bb9e3976aebcd2"
format_stamp: "Formatted at 2026-04-29 14:03:38 on dist-test-slave-8n77"
server_key: "a90227d39f1d6dcf8d7fee7a38426d4f"
server_key_iv: "11056c5ea4a2278d0210e24b1592ea01"
server_key_version: "encryptionkey@0"
I20260429 14:03:38.809984 3389 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:03:38.819944 3389 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:03:38.822362 3389 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:03:38.822515 3389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:03:38.826934 3389 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:42069
I20260429 14:03:38.826977 3456 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:42069 every 8 connection(s)
I20260429 14:03:38.827332 3389 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:03:38.827934 3457 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:03:38.830411 3457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b0390cda90247b3a4bb9e3976aebcd2: Bootstrap starting.
I20260429 14:03:38.831045 3457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b0390cda90247b3a4bb9e3976aebcd2: Neither blocks nor log segments found. Creating new log.
I20260429 14:03:38.831306 3457 log.cc:826] T 00000000000000000000000000000000 P 8b0390cda90247b3a4bb9e3976aebcd2: Log is configured to *not* fsync() on all Append() calls
I20260429 14:03:38.831594 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 3389
I20260429 14:03:38.831666 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:03:38.831857 2839 external_mini_cluster.cc:1468] Setting key 83280df9b53747e5a755c45012684765
I20260429 14:03:38.832132 3457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b0390cda90247b3a4bb9e3976aebcd2: No bootstrap required, opened a new log
I20260429 14:03:38.834223 3457 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8b0390cda90247b3a4bb9e3976aebcd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b0390cda90247b3a4bb9e3976aebcd2" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 42069 } }
I20260429 14:03:38.834714 3457 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8b0390cda90247b3a4bb9e3976aebcd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:03:38.834774 3457 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8b0390cda90247b3a4bb9e3976aebcd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b0390cda90247b3a4bb9e3976aebcd2, State: Initialized, Role: FOLLOWER
I20260429 14:03:38.834909 3457 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b0390cda90247b3a4bb9e3976aebcd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b0390cda90247b3a4bb9e3976aebcd2" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 42069 } }
I20260429 14:03:38.834980 3457 raft_consensus.cc:399] T 00000000000000000000000000000000 P 8b0390cda90247b3a4bb9e3976aebcd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:03:38.835018 3457 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8b0390cda90247b3a4bb9e3976aebcd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:03:38.835064 3457 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8b0390cda90247b3a4bb9e3976aebcd2 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:03:38.835850 3457 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8b0390cda90247b3a4bb9e3976aebcd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b0390cda90247b3a4bb9e3976aebcd2" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 42069 } }
I20260429 14:03:38.835966 3457 leader_election.cc:304] T 00000000000000000000000000000000 P 8b0390cda90247b3a4bb9e3976aebcd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b0390cda90247b3a4bb9e3976aebcd2; no voters:
I20260429 14:03:38.836297 3457 leader_election.cc:290] T 00000000000000000000000000000000 P 8b0390cda90247b3a4bb9e3976aebcd2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:03:38.836325 3462 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8b0390cda90247b3a4bb9e3976aebcd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:03:38.836442 3462 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8b0390cda90247b3a4bb9e3976aebcd2 [term 1 LEADER]: Becoming Leader. State: Replica: 8b0390cda90247b3a4bb9e3976aebcd2, State: Running, Role: LEADER
I20260429 14:03:38.836556 3462 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b0390cda90247b3a4bb9e3976aebcd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b0390cda90247b3a4bb9e3976aebcd2" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 42069 } }
I20260429 14:03:38.836623 3457 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8b0390cda90247b3a4bb9e3976aebcd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:03:38.836900 3462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b0390cda90247b3a4bb9e3976aebcd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b0390cda90247b3a4bb9e3976aebcd2. Latest consensus state: current_term: 1 leader_uuid: "8b0390cda90247b3a4bb9e3976aebcd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b0390cda90247b3a4bb9e3976aebcd2" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 42069 } } }
I20260429 14:03:38.836982 3462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b0390cda90247b3a4bb9e3976aebcd2 [sys.catalog]: This master's current role is: LEADER
I20260429 14:03:38.837092 3462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b0390cda90247b3a4bb9e3976aebcd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b0390cda90247b3a4bb9e3976aebcd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b0390cda90247b3a4bb9e3976aebcd2" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 42069 } } }
I20260429 14:03:38.837154 3462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b0390cda90247b3a4bb9e3976aebcd2 [sys.catalog]: This master's current role is: LEADER
I20260429 14:03:38.837206 3466 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:03:38.837990 3466 catalog_manager.cc:1494] Initializing Kudu cluster ID...
Apr 29 14:03:38 dist-test-slave-8n77 krb5kdc[3373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:03:38.840677 3466 catalog_manager.cc:1357] Generated new cluster ID: eb0d52233a0243e99c1bfaec2bce3cd5
I20260429 14:03:38.840742 3466 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:03:38.845278 3461 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:38.832447 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33402 (local address 127.2.197.254:42069)
0429 14:03:38.832644 (+ 197us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:38.832655 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:38.832680 (+ 25us) server_negotiation.cc:408] Connection header received
0429 14:03:38.833273 (+ 593us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:38.833283 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:38.833336 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:38.833472 (+ 136us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:38.834396 (+ 924us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:38.835775 (+ 1379us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:38.836538 (+ 763us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:38.836793 (+ 255us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:38.840164 (+ 3371us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:38.840173 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:38.840174 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:38.840187 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:38.843850 (+ 3663us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:38.844298 (+ 448us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:38.844301 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:38.844301 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:38.844354 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:38.844525 (+ 171us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:38.844526 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:38.844526 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:38.844728 (+ 202us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:38.844883 (+ 155us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:38.844996 (+ 113us) server_negotiation.cc:300] Negotiation successful
0429 14:03:38.845129 (+ 133us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":108,"threads_started":1}
I20260429 14:03:38.861400 3466 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:03:38.862207 3466 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:03:38.876173 3466 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 8b0390cda90247b3a4bb9e3976aebcd2: Generated new TSK 0
I20260429 14:03:38.876384 3466 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:03:38.940331 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.193:0
--local_ip_for_outbound_sockets=127.2.197.193
--webserver_interface=127.2.197.193
--webserver_port=0
--tserver_master_addrs=127.2.197.254:42069
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:33077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:03:39.020617 3485 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:03:39.020869 3485 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:03:39.020910 3485 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:03:39.022658 3485 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:03:39.022743 3485 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:03:39.022799 3485 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.193
I20260429 14:03:39.024672 3485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:33077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.197.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:42069
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.193
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:03:39.024983 3485 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:03:39.025194 3485 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:03:39.028252 3490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:03:39.028235 3491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:03:39.028235 3493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:03:39.028506 3485 server_base.cc:1061] running on GCE node
I20260429 14:03:39.028710 3485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:03:39.028982 3485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:03:39.030176 3485 hybrid_clock.cc:648] HybridClock initialized: now 1777471419030121 us; error 78 us; skew 500 ppm
Apr 29 14:03:39 dist-test-slave-8n77 krb5kdc[3373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471419, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:03:39.033175 3485 init.cc:377] Logged in from keytab as kudu/127.2.197.193@KRBTEST.COM (short username kudu)
I20260429 14:03:39.033773 3485 webserver.cc:492] Webserver started at http://127.2.197.193:34715/ using document root <none> and password file <none>
I20260429 14:03:39.033995 3485 fs_manager.cc:362] Metadata directory not provided
I20260429 14:03:39.034061 3485 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:03:39.034163 3485 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:03:39.035167 3485 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-0/data/instance:
uuid: "474d8c15499f425f939fa1fd858d56c4"
format_stamp: "Formatted at 2026-04-29 14:03:39 on dist-test-slave-8n77"
server_key: "9a3f03bb3f07db2fe2b506c16fa4016e"
server_key_iv: "b1ac2dcb5c1289a7c837b417f45ac033"
server_key_version: "encryptionkey@0"
I20260429 14:03:39.035522 3485 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance:
uuid: "474d8c15499f425f939fa1fd858d56c4"
format_stamp: "Formatted at 2026-04-29 14:03:39 on dist-test-slave-8n77"
server_key: "9a3f03bb3f07db2fe2b506c16fa4016e"
server_key_iv: "b1ac2dcb5c1289a7c837b417f45ac033"
server_key_version: "encryptionkey@0"
I20260429 14:03:39.036866 3485 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260429 14:03:39.037743 3500 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:39.037968 3485 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:03:39.038054 3485 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-0/wal
uuid: "474d8c15499f425f939fa1fd858d56c4"
format_stamp: "Formatted at 2026-04-29 14:03:39 on dist-test-slave-8n77"
server_key: "9a3f03bb3f07db2fe2b506c16fa4016e"
server_key_iv: "b1ac2dcb5c1289a7c837b417f45ac033"
server_key_version: "encryptionkey@0"
I20260429 14:03:39.038146 3485 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:03:39.053061 3485 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:03:39.055213 3485 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:03:39.055408 3485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:03:39.055675 3485 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:03:39.056026 3485 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:03:39.056092 3485 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:39.056159 3485 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:03:39.056187 3485 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:39.062919 3485 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.193:38115
I20260429 14:03:39.063287 3485 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
I20260429 14:03:39.063098 3613 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.193:38115 every 8 connection(s)
I20260429 14:03:39.066077 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 3485
I20260429 14:03:39.066190 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance
I20260429 14:03:39.066416 2839 external_mini_cluster.cc:1468] Setting key b0152991152df105c89f2ceb458e2b44
Apr 29 14:03:39 dist-test-slave-8n77 krb5kdc[3373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471419, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:03:39.074698 3461 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:39.064122 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:55657 (local address 127.2.197.254:42069)
0429 14:03:39.064234 (+ 112us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:39.064237 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:39.064503 (+ 266us) server_negotiation.cc:408] Connection header received
0429 14:03:39.065077 (+ 574us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:39.065078 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:39.065090 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:39.065155 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:39.065894 (+ 739us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.066896 (+ 1002us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:39.067618 (+ 722us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.067842 (+ 224us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:39.071174 (+ 3332us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:39.071181 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:39.071181 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:39.071192 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:39.073073 (+ 1881us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:39.073601 (+ 528us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:39.073602 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:39.073603 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:39.073637 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:39.073915 (+ 278us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:39.073916 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:39.073916 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:39.074023 (+ 107us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:39.074094 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:39.074500 (+ 406us) server_negotiation.cc:300] Negotiation successful
0429 14:03:39.074629 (+ 129us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260429 14:03:39.074842 3616 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:39.064115 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:42069 (local address 127.2.197.193:55657)
0429 14:03:39.064302 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:39.064341 (+ 39us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:39.064971 (+ 630us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:39.065210 (+ 239us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:39.065221 (+ 11us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:39.065518 (+ 297us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:03:39.065774 (+ 256us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:39.065777 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.067029 (+ 1252us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:39.067031 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:39.067474 (+ 443us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:39.067476 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.067618 (+ 142us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:39.069129 (+ 1511us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:03:39.069144 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:03:39.070930 (+ 1786us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:03:39.073179 (+ 2249us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:39.073187 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:39.073190 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:39.073495 (+ 305us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:39.073718 (+ 223us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:39.073719 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:39.073720 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:39.073822 (+ 102us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:39.074105 (+ 283us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:03:39.074110 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:03:39.074310 (+ 200us) client_negotiation.cc:770] Sending connection context
0429 14:03:39.074444 (+ 134us) client_negotiation.cc:241] Negotiation successful
0429 14:03:39.074603 (+ 159us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":83,"threads_started":1}
I20260429 14:03:39.075366 3614 heartbeater.cc:344] Connected to a master server at 127.2.197.254:42069
I20260429 14:03:39.075457 3614 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for kudu/127.2.197.194@KRBTEST.COM; defaulting to no policy
I20260429 14:03:39.075663 3614 heartbeater.cc:507] Master 127.2.197.254:42069 requested a full tablet report, sending...
I20260429 14:03:39.076117 3421 ts_manager.cc:194] Registered new tserver with Master: 474d8c15499f425f939fa1fd858d56c4 (127.2.197.193:38115)
I20260429 14:03:39.076773 3421 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.193@KRBTEST.COM'} at 127.2.197.193:55657
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:03:39.122836 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.194:0
--local_ip_for_outbound_sockets=127.2.197.194
--webserver_interface=127.2.197.194
--webserver_port=0
--tserver_master_addrs=127.2.197.254:42069
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:33077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:03:39.201875 3621 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:03:39.202075 3621 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:03:39.202102 3621 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:03:39.203825 3621 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:03:39.203896 3621 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:03:39.203928 3621 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.194
I20260429 14:03:39.205690 3621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:33077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.197.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:42069
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.194
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:03:39.205956 3621 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:03:39.206169 3621 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:03:39.209077 3626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:03:39.209079 3627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:03:39.209079 3621 server_base.cc:1061] running on GCE node
W20260429 14:03:39.209079 3629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:03:39.209592 3621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:03:39.209900 3621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:03:39.211069 3621 hybrid_clock.cc:648] HybridClock initialized: now 1777471419211034 us; error 47 us; skew 500 ppm
Apr 29 14:03:39 dist-test-slave-8n77 krb5kdc[3373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471419, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:03:39.213581 3621 init.cc:377] Logged in from keytab as kudu/127.2.197.194@KRBTEST.COM (short username kudu)
I20260429 14:03:39.214202 3621 webserver.cc:492] Webserver started at http://127.2.197.194:34485/ using document root <none> and password file <none>
I20260429 14:03:39.214427 3621 fs_manager.cc:362] Metadata directory not provided
I20260429 14:03:39.214488 3621 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:03:39.214696 3621 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:03:39.216168 3621 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-1/data/instance:
uuid: "35ad592ec35e473cbfe41566dad67848"
format_stamp: "Formatted at 2026-04-29 14:03:39 on dist-test-slave-8n77"
server_key: "69af4bc402da75478885d17af810b2b5"
server_key_iv: "1db3f61b02231f9b0661794930866b5f"
server_key_version: "encryptionkey@0"
I20260429 14:03:39.216764 3621 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance:
uuid: "35ad592ec35e473cbfe41566dad67848"
format_stamp: "Formatted at 2026-04-29 14:03:39 on dist-test-slave-8n77"
server_key: "69af4bc402da75478885d17af810b2b5"
server_key_iv: "1db3f61b02231f9b0661794930866b5f"
server_key_version: "encryptionkey@0"
I20260429 14:03:39.218669 3621 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260429 14:03:39.219684 3636 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:39.220048 3621 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:03:39.220163 3621 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-1/wal
uuid: "35ad592ec35e473cbfe41566dad67848"
format_stamp: "Formatted at 2026-04-29 14:03:39 on dist-test-slave-8n77"
server_key: "69af4bc402da75478885d17af810b2b5"
server_key_iv: "1db3f61b02231f9b0661794930866b5f"
server_key_version: "encryptionkey@0"
I20260429 14:03:39.220264 3621 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:03:39.253168 3621 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:03:39.255335 3621 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:03:39.255511 3621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:03:39.255748 3621 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:03:39.256073 3621 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:03:39.256129 3621 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:39.256170 3621 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:03:39.256214 3621 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:39.262827 3621 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.194:37585
I20260429 14:03:39.262923 3749 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.194:37585 every 8 connection(s)
I20260429 14:03:39.263231 3621 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
I20260429 14:03:39.268576 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 3621
I20260429 14:03:39.268669 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance
I20260429 14:03:39.268857 2839 external_mini_cluster.cc:1468] Setting key 438561ee28f05f6da2affb50d23a989f
Apr 29 14:03:39 dist-test-slave-8n77 krb5kdc[3373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471419, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:03:39.273087 3461 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:39.264160 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:43397 (local address 127.2.197.254:42069)
0429 14:03:39.264234 (+ 74us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:39.264237 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:39.264406 (+ 169us) server_negotiation.cc:408] Connection header received
0429 14:03:39.264912 (+ 506us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:39.264913 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:39.264924 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:39.264990 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:39.265743 (+ 753us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.266236 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:39.267007 (+ 771us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.267139 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:39.269837 (+ 2698us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:39.269841 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:39.269842 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:39.269850 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:39.271514 (+ 1664us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:39.271958 (+ 444us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:39.271959 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:39.271960 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:39.271991 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:39.272195 (+ 204us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:39.272196 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:39.272197 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:39.272302 (+ 105us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:39.272380 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:39.272888 (+ 508us) server_negotiation.cc:300] Negotiation successful
0429 14:03:39.272992 (+ 104us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260429 14:03:39.273202 3752 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:39.264144 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:42069 (local address 127.2.197.194:43397)
0429 14:03:39.264283 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:39.264319 (+ 36us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:39.264810 (+ 491us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:39.265031 (+ 221us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:39.265034 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:39.265345 (+ 311us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:03:39.265601 (+ 256us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:39.265604 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.266490 (+ 886us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:39.266492 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:39.266908 (+ 416us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:39.266909 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.267006 (+ 97us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:39.267976 (+ 970us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:03:39.267985 (+ 9us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:03:39.269737 (+ 1752us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:03:39.271634 (+ 1897us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:39.271640 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:39.271643 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:39.271863 (+ 220us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:39.272058 (+ 195us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:39.272059 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:39.272059 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:39.272129 (+ 70us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:39.272369 (+ 240us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:03:39.272374 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:03:39.272504 (+ 130us) client_negotiation.cc:770] Sending connection context
0429 14:03:39.272845 (+ 341us) client_negotiation.cc:241] Negotiation successful
0429 14:03:39.273004 (+ 159us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":76,"threads_started":1}
I20260429 14:03:39.273676 3750 heartbeater.cc:344] Connected to a master server at 127.2.197.254:42069
I20260429 14:03:39.273767 3750 heartbeater.cc:461] Registering TS with master...
I20260429 14:03:39.273973 3750 heartbeater.cc:507] Master 127.2.197.254:42069 requested a full tablet report, sending...
I20260429 14:03:39.274545 3421 ts_manager.cc:194] Registered new tserver with Master: 35ad592ec35e473cbfe41566dad67848 (127.2.197.194:37585)
I20260429 14:03:39.275061 3421 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.194@KRBTEST.COM'} at 127.2.197.194:43397
WARNING: no policy specified for kudu/127.2.197.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:03:39.327699 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.195:0
--local_ip_for_outbound_sockets=127.2.197.195
--webserver_interface=127.2.197.195
--webserver_port=0
--tserver_master_addrs=127.2.197.254:42069
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:33077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:03:39.409124 3757 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:03:39.409390 3757 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:03:39.409435 3757 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:03:39.411290 3757 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:03:39.411375 3757 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:03:39.411422 3757 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.195
I20260429 14:03:39.413210 3757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:33077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.197.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:42069
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.195
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:03:39.413524 3757 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:03:39.413759 3757 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:03:39.417039 3763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:03:39.417259 3762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:03:39.417275 3765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:03:39.417452 3757 server_base.cc:1061] running on GCE node
I20260429 14:03:39.417855 3757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:03:39.418051 3757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:03:39.419207 3757 hybrid_clock.cc:648] HybridClock initialized: now 1777471419419176 us; error 47 us; skew 500 ppm
Apr 29 14:03:39 dist-test-slave-8n77 krb5kdc[3373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471419, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:03:39.421877 3757 init.cc:377] Logged in from keytab as kudu/127.2.197.195@KRBTEST.COM (short username kudu)
I20260429 14:03:39.422494 3757 webserver.cc:492] Webserver started at http://127.2.197.195:42785/ using document root <none> and password file <none>
I20260429 14:03:39.422693 3757 fs_manager.cc:362] Metadata directory not provided
I20260429 14:03:39.422770 3757 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:03:39.423020 3757 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:03:39.424501 3757 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-2/data/instance:
uuid: "ba1a455792e44812a83f24dd6f74464c"
format_stamp: "Formatted at 2026-04-29 14:03:39 on dist-test-slave-8n77"
server_key: "c4fcc2b0c2f88d210221381c411595a0"
server_key_iv: "ea31d0b3fac5c37c8e0f026d05bae354"
server_key_version: "encryptionkey@0"
I20260429 14:03:39.424871 3757 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance:
uuid: "ba1a455792e44812a83f24dd6f74464c"
format_stamp: "Formatted at 2026-04-29 14:03:39 on dist-test-slave-8n77"
server_key: "c4fcc2b0c2f88d210221381c411595a0"
server_key_iv: "ea31d0b3fac5c37c8e0f026d05bae354"
server_key_version: "encryptionkey@0"
I20260429 14:03:39.426505 3757 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260429 14:03:39.427361 3772 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:39.427678 3757 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:03:39.427868 3757 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-2/wal
uuid: "ba1a455792e44812a83f24dd6f74464c"
format_stamp: "Formatted at 2026-04-29 14:03:39 on dist-test-slave-8n77"
server_key: "c4fcc2b0c2f88d210221381c411595a0"
server_key_iv: "ea31d0b3fac5c37c8e0f026d05bae354"
server_key_version: "encryptionkey@0"
I20260429 14:03:39.427954 3757 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:03:39.451964 3757 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:03:39.454782 3757 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:03:39.455057 3757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:03:39.455382 3757 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:03:39.455919 3757 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:03:39.455981 3757 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:39.456007 3757 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:03:39.456020 3757 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:39.462455 3757 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.195:45473
I20260429 14:03:39.462627 3885 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.195:45473 every 8 connection(s)
I20260429 14:03:39.462877 3757 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
Apr 29 14:03:39 dist-test-slave-8n77 krb5kdc[3373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471419, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:03:39.473125 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 3757
I20260429 14:03:39.473244 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance
I20260429 14:03:39.473237 3461 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:39.463935 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:36519 (local address 127.2.197.254:42069)
0429 14:03:39.464017 (+ 82us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:39.464019 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:39.464484 (+ 465us) server_negotiation.cc:408] Connection header received
0429 14:03:39.465089 (+ 605us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:39.465091 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:39.465104 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:39.465167 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:39.465954 (+ 787us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.466501 (+ 547us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:39.467071 (+ 570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.467236 (+ 165us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:39.469789 (+ 2553us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:39.469794 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:39.469795 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:39.469803 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:39.471498 (+ 1695us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:39.472132 (+ 634us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:39.472134 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:39.472135 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:39.472171 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:39.472476 (+ 305us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:39.472477 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:39.472477 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:39.472620 (+ 143us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:39.472679 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:39.472997 (+ 318us) server_negotiation.cc:300] Negotiation successful
0429 14:03:39.473111 (+ 114us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260429 14:03:39.473278 3888 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:39.463903 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:42069 (local address 127.2.197.195:36519)
0429 14:03:39.464335 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:39.464367 (+ 32us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:39.464971 (+ 604us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:39.465243 (+ 272us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:39.465247 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:39.465566 (+ 319us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:03:39.465834 (+ 268us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:39.465837 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.466610 (+ 773us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:39.466611 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:39.466986 (+ 375us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:39.466988 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.467121 (+ 133us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:39.468093 (+ 972us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:03:39.468102 (+ 9us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:03:39.469670 (+ 1568us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:03:39.471692 (+ 2022us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:39.471701 (+ 9us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:39.471704 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:39.472000 (+ 296us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:39.472299 (+ 299us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:39.472302 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:39.472302 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:39.472406 (+ 104us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:39.472683 (+ 277us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:03:39.472686 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:03:39.472818 (+ 132us) client_negotiation.cc:770] Sending connection context
0429 14:03:39.472920 (+ 102us) client_negotiation.cc:241] Negotiation successful
0429 14:03:39.473064 (+ 144us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":326,"threads_started":1}
I20260429 14:03:39.473433 2839 external_mini_cluster.cc:1468] Setting key eed6e89ae8d2a70b280b12366b3fbf8a
I20260429 14:03:39.473812 3886 heartbeater.cc:344] Connected to a master server at 127.2.197.254:42069
I20260429 14:03:39.473938 3886 heartbeater.cc:461] Registering TS with master...
I20260429 14:03:39.474167 3886 heartbeater.cc:507] Master 127.2.197.254:42069 requested a full tablet report, sending...
I20260429 14:03:39.474570 3421 ts_manager.cc:194] Registered new tserver with Master: ba1a455792e44812a83f24dd6f74464c (127.2.197.195:45473)
I20260429 14:03:39.475077 3421 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.195@KRBTEST.COM'} at 127.2.197.195:36519
I20260429 14:03:39.486018 2839 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260429 14:03:39.494880 3461 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:39.488002 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33412 (local address 127.2.197.254:42069)
0429 14:03:39.488079 (+ 77us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:39.488082 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:39.488092 (+ 10us) server_negotiation.cc:408] Connection header received
0429 14:03:39.488156 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:39.488157 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:39.488305 (+ 148us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:39.488477 (+ 172us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:39.489227 (+ 750us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.489712 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:39.490386 (+ 674us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.490599 (+ 213us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:39.491925 (+ 1326us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:39.491929 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:39.491930 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:39.491938 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:39.493688 (+ 1750us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:39.494082 (+ 394us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:39.494083 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:39.494084 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:39.494114 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:39.494310 (+ 196us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:39.494311 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:39.494312 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:39.494446 (+ 134us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:39.494565 (+ 119us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:39.494686 (+ 121us) server_negotiation.cc:300] Negotiation successful
0429 14:03:39.494773 (+ 87us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260429 14:03:39.498076 3421 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 14:03:39.499825 3421 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 14:03:39.506088 3898 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:39.503117 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:37585 (local address 127.0.0.1:36578)
0429 14:03:39.503457 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:39.503469 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:39.503538 (+ 69us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:39.503762 (+ 224us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:39.503765 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:39.503768 (+ 3us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:03:39.504054 (+ 286us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:39.504057 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.505048 (+ 991us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:39.505049 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:39.505812 (+ 763us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:39.505813 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.505882 (+ 69us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:39.505918 (+ 36us) client_negotiation.cc:770] Sending connection context
0429 14:03:39.505967 (+ 49us) client_negotiation.cc:241] Negotiation successful
0429 14:03:39.506005 (+ 38us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":80,"threads_started":1}
I20260429 14:03:39.506620 3900 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:39.503307 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:45473 (local address 127.0.0.1:48164)
0429 14:03:39.503582 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:39.503592 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:39.503668 (+ 76us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:39.503952 (+ 284us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:39.503953 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:39.503955 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:03:39.504180 (+ 225us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:39.504182 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.505506 (+ 1324us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:39.505507 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:39.506412 (+ 905us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:39.506414 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.506477 (+ 63us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:39.506481 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 14:03:39.506513 (+ 32us) client_negotiation.cc:241] Negotiation successful
0429 14:03:39.506552 (+ 39us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":77,"threads_started":1}
I20260429 14:03:39.507146 3901 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:39.503311 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36578 (local address 127.2.197.194:37585)
0429 14:03:39.503462 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:39.503472 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:39.503589 (+ 117us) server_negotiation.cc:408] Connection header received
0429 14:03:39.503634 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:39.503637 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:39.503654 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:39.503750 (+ 96us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:03:39.504138 (+ 388us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.504953 (+ 815us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:39.506363 (+ 1410us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.506945 (+ 582us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:39.507006 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:39.507035 (+ 29us) server_negotiation.cc:300] Negotiation successful
0429 14:03:39.507076 (+ 41us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":82,"threads_started":1}
I20260429 14:03:39.507413 3897 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:39.503107 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:38115 (local address 127.0.0.1:48962)
0429 14:03:39.503272 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:39.503304 (+ 32us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:39.503380 (+ 76us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:39.504130 (+ 750us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:39.504131 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:39.504132 (+ 1us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:03:39.504324 (+ 192us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:39.504325 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.505798 (+ 1473us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:39.505799 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:39.507133 (+ 1334us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:39.507135 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.507232 (+ 97us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:39.507238 (+ 6us) client_negotiation.cc:770] Sending connection context
0429 14:03:39.507272 (+ 34us) client_negotiation.cc:241] Negotiation successful
0429 14:03:39.507311 (+ 39us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":71,"threads_started":1}
I20260429 14:03:39.507553 3902 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:39.503539 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48164 (local address 127.2.197.195:45473)
0429 14:03:39.503719 (+ 180us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:39.503726 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:39.503735 (+ 9us) server_negotiation.cc:408] Connection header received
0429 14:03:39.503776 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:39.503796 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:39.503811 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:39.503873 (+ 62us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:03:39.504257 (+ 384us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.505417 (+ 1160us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:39.506840 (+ 1423us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.507376 (+ 536us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:39.507417 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:39.507445 (+ 28us) server_negotiation.cc:300] Negotiation successful
0429 14:03:39.507485 (+ 40us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":88,"threads_started":1}
I20260429 14:03:39.507726 3684 tablet_service.cc:1511] Processing CreateTablet for tablet 10738ffeb2314b858bb69ab5ec68fbd4 (DEFAULT_TABLE table=test-table [id=d2d1d7f4226247d9bd4a02e8d81be48d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:03:39.508041 3684 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10738ffeb2314b858bb69ab5ec68fbd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:03:39.508062 3820 tablet_service.cc:1511] Processing CreateTablet for tablet 10738ffeb2314b858bb69ab5ec68fbd4 (DEFAULT_TABLE table=test-table [id=d2d1d7f4226247d9bd4a02e8d81be48d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:03:39.508287 3820 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10738ffeb2314b858bb69ab5ec68fbd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:03:39.508793 3899 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:39.503235 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48962 (local address 127.2.197.193:38115)
0429 14:03:39.503913 (+ 678us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:39.503920 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:39.503930 (+ 10us) server_negotiation.cc:408] Connection header received
0429 14:03:39.503967 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:39.503969 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:39.503983 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:39.504050 (+ 67us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:03:39.504405 (+ 355us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.505620 (+ 1215us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:39.507398 (+ 1778us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.508639 (+ 1241us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:39.508671 (+ 32us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:39.508707 (+ 36us) server_negotiation.cc:300] Negotiation successful
0429 14:03:39.508737 (+ 30us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":95,"threads_started":1}
I20260429 14:03:39.509264 3548 tablet_service.cc:1511] Processing CreateTablet for tablet 10738ffeb2314b858bb69ab5ec68fbd4 (DEFAULT_TABLE table=test-table [id=d2d1d7f4226247d9bd4a02e8d81be48d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:03:39.509523 3548 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10738ffeb2314b858bb69ab5ec68fbd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:03:39.511570 3903 tablet_bootstrap.cc:492] T 10738ffeb2314b858bb69ab5ec68fbd4 P 35ad592ec35e473cbfe41566dad67848: Bootstrap starting.
I20260429 14:03:39.511973 3904 tablet_bootstrap.cc:492] T 10738ffeb2314b858bb69ab5ec68fbd4 P ba1a455792e44812a83f24dd6f74464c: Bootstrap starting.
I20260429 14:03:39.512310 3903 tablet_bootstrap.cc:654] T 10738ffeb2314b858bb69ab5ec68fbd4 P 35ad592ec35e473cbfe41566dad67848: Neither blocks nor log segments found. Creating new log.
I20260429 14:03:39.512516 3904 tablet_bootstrap.cc:654] T 10738ffeb2314b858bb69ab5ec68fbd4 P ba1a455792e44812a83f24dd6f74464c: Neither blocks nor log segments found. Creating new log.
I20260429 14:03:39.512684 3903 log.cc:826] T 10738ffeb2314b858bb69ab5ec68fbd4 P 35ad592ec35e473cbfe41566dad67848: Log is configured to *not* fsync() on all Append() calls
I20260429 14:03:39.512881 3904 log.cc:826] T 10738ffeb2314b858bb69ab5ec68fbd4 P ba1a455792e44812a83f24dd6f74464c: Log is configured to *not* fsync() on all Append() calls
I20260429 14:03:39.513288 3907 tablet_bootstrap.cc:492] T 10738ffeb2314b858bb69ab5ec68fbd4 P 474d8c15499f425f939fa1fd858d56c4: Bootstrap starting.
I20260429 14:03:39.513473 3903 tablet_bootstrap.cc:492] T 10738ffeb2314b858bb69ab5ec68fbd4 P 35ad592ec35e473cbfe41566dad67848: No bootstrap required, opened a new log
I20260429 14:03:39.513550 3903 ts_tablet_manager.cc:1403] T 10738ffeb2314b858bb69ab5ec68fbd4 P 35ad592ec35e473cbfe41566dad67848: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260429 14:03:39.513928 3904 tablet_bootstrap.cc:492] T 10738ffeb2314b858bb69ab5ec68fbd4 P ba1a455792e44812a83f24dd6f74464c: No bootstrap required, opened a new log
I20260429 14:03:39.514014 3904 ts_tablet_manager.cc:1403] T 10738ffeb2314b858bb69ab5ec68fbd4 P ba1a455792e44812a83f24dd6f74464c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260429 14:03:39.515386 3903 raft_consensus.cc:359] T 10738ffeb2314b858bb69ab5ec68fbd4 P 35ad592ec35e473cbfe41566dad67848 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ad592ec35e473cbfe41566dad67848" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37585 } } peers { permanent_uuid: "474d8c15499f425f939fa1fd858d56c4" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 38115 } } peers { permanent_uuid: "ba1a455792e44812a83f24dd6f74464c" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 45473 } }
I20260429 14:03:39.515645 3903 raft_consensus.cc:385] T 10738ffeb2314b858bb69ab5ec68fbd4 P 35ad592ec35e473cbfe41566dad67848 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:03:39.515690 3903 raft_consensus.cc:740] T 10738ffeb2314b858bb69ab5ec68fbd4 P 35ad592ec35e473cbfe41566dad67848 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35ad592ec35e473cbfe41566dad67848, State: Initialized, Role: FOLLOWER
I20260429 14:03:39.515767 3904 raft_consensus.cc:359] T 10738ffeb2314b858bb69ab5ec68fbd4 P ba1a455792e44812a83f24dd6f74464c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ad592ec35e473cbfe41566dad67848" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37585 } } peers { permanent_uuid: "474d8c15499f425f939fa1fd858d56c4" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 38115 } } peers { permanent_uuid: "ba1a455792e44812a83f24dd6f74464c" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 45473 } }
I20260429 14:03:39.515826 3903 consensus_queue.cc:260] T 10738ffeb2314b858bb69ab5ec68fbd4 P 35ad592ec35e473cbfe41566dad67848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ad592ec35e473cbfe41566dad67848" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37585 } } peers { permanent_uuid: "474d8c15499f425f939fa1fd858d56c4" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 38115 } } peers { permanent_uuid: "ba1a455792e44812a83f24dd6f74464c" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 45473 } }
I20260429 14:03:39.515893 3904 raft_consensus.cc:385] T 10738ffeb2314b858bb69ab5ec68fbd4 P ba1a455792e44812a83f24dd6f74464c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:03:39.515925 3904 raft_consensus.cc:740] T 10738ffeb2314b858bb69ab5ec68fbd4 P ba1a455792e44812a83f24dd6f74464c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba1a455792e44812a83f24dd6f74464c, State: Initialized, Role: FOLLOWER
I20260429 14:03:39.516036 3904 consensus_queue.cc:260] T 10738ffeb2314b858bb69ab5ec68fbd4 P ba1a455792e44812a83f24dd6f74464c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ad592ec35e473cbfe41566dad67848" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37585 } } peers { permanent_uuid: "474d8c15499f425f939fa1fd858d56c4" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 38115 } } peers { permanent_uuid: "ba1a455792e44812a83f24dd6f74464c" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 45473 } }
I20260429 14:03:39.516109 3903 ts_tablet_manager.cc:1434] T 10738ffeb2314b858bb69ab5ec68fbd4 P 35ad592ec35e473cbfe41566dad67848: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260429 14:03:39.516249 3750 heartbeater.cc:499] Master 127.2.197.254:42069 was elected leader, sending a full tablet report...
I20260429 14:03:39.516289 3904 ts_tablet_manager.cc:1434] T 10738ffeb2314b858bb69ab5ec68fbd4 P ba1a455792e44812a83f24dd6f74464c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260429 14:03:39.516336 3886 heartbeater.cc:499] Master 127.2.197.254:42069 was elected leader, sending a full tablet report...
I20260429 14:03:39.516732 3907 tablet_bootstrap.cc:654] T 10738ffeb2314b858bb69ab5ec68fbd4 P 474d8c15499f425f939fa1fd858d56c4: Neither blocks nor log segments found. Creating new log.
I20260429 14:03:39.517071 3907 log.cc:826] T 10738ffeb2314b858bb69ab5ec68fbd4 P 474d8c15499f425f939fa1fd858d56c4: Log is configured to *not* fsync() on all Append() calls
I20260429 14:03:39.518177 3907 tablet_bootstrap.cc:492] T 10738ffeb2314b858bb69ab5ec68fbd4 P 474d8c15499f425f939fa1fd858d56c4: No bootstrap required, opened a new log
I20260429 14:03:39.518246 3907 ts_tablet_manager.cc:1403] T 10738ffeb2314b858bb69ab5ec68fbd4 P 474d8c15499f425f939fa1fd858d56c4: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260429 14:03:39.519717 3907 raft_consensus.cc:359] T 10738ffeb2314b858bb69ab5ec68fbd4 P 474d8c15499f425f939fa1fd858d56c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ad592ec35e473cbfe41566dad67848" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37585 } } peers { permanent_uuid: "474d8c15499f425f939fa1fd858d56c4" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 38115 } } peers { permanent_uuid: "ba1a455792e44812a83f24dd6f74464c" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 45473 } }
I20260429 14:03:39.519836 3907 raft_consensus.cc:385] T 10738ffeb2314b858bb69ab5ec68fbd4 P 474d8c15499f425f939fa1fd858d56c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:03:39.519866 3907 raft_consensus.cc:740] T 10738ffeb2314b858bb69ab5ec68fbd4 P 474d8c15499f425f939fa1fd858d56c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 474d8c15499f425f939fa1fd858d56c4, State: Initialized, Role: FOLLOWER
I20260429 14:03:39.519974 3907 consensus_queue.cc:260] T 10738ffeb2314b858bb69ab5ec68fbd4 P 474d8c15499f425f939fa1fd858d56c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ad592ec35e473cbfe41566dad67848" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37585 } } peers { permanent_uuid: "474d8c15499f425f939fa1fd858d56c4" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 38115 } } peers { permanent_uuid: "ba1a455792e44812a83f24dd6f74464c" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 45473 } }
I20260429 14:03:39.520222 3907 ts_tablet_manager.cc:1434] T 10738ffeb2314b858bb69ab5ec68fbd4 P 474d8c15499f425f939fa1fd858d56c4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260429 14:03:39.520380 3614 heartbeater.cc:499] Master 127.2.197.254:42069 was elected leader, sending a full tablet report...
W20260429 14:03:39.564172 3615 tablet.cc:2404] T 10738ffeb2314b858bb69ab5ec68fbd4 P 474d8c15499f425f939fa1fd858d56c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260429 14:03:39.713469 3887 tablet.cc:2404] T 10738ffeb2314b858bb69ab5ec68fbd4 P ba1a455792e44812a83f24dd6f74464c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 14:03:39.756017 3908 raft_consensus.cc:493] T 10738ffeb2314b858bb69ab5ec68fbd4 P 35ad592ec35e473cbfe41566dad67848 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:03:39.756168 3908 raft_consensus.cc:515] T 10738ffeb2314b858bb69ab5ec68fbd4 P 35ad592ec35e473cbfe41566dad67848 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ad592ec35e473cbfe41566dad67848" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37585 } } peers { permanent_uuid: "474d8c15499f425f939fa1fd858d56c4" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 38115 } } peers { permanent_uuid: "ba1a455792e44812a83f24dd6f74464c" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 45473 } }
I20260429 14:03:39.756554 3908 leader_election.cc:290] T 10738ffeb2314b858bb69ab5ec68fbd4 P 35ad592ec35e473cbfe41566dad67848 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 474d8c15499f425f939fa1fd858d56c4 (127.2.197.193:38115), ba1a455792e44812a83f24dd6f74464c (127.2.197.195:45473)
I20260429 14:03:39.759177 3752 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:39.756747 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:38115 (local address 127.2.197.194:37339)
0429 14:03:39.756855 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:39.756869 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:39.756916 (+ 47us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:39.757096 (+ 180us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:39.757097 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:39.757099 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:03:39.757355 (+ 256us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:39.757357 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.758265 (+ 908us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:39.758266 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:39.759008 (+ 742us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:39.759010 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.759072 (+ 62us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:39.759075 (+ 3us) client_negotiation.cc:770] Sending connection context
0429 14:03:39.759098 (+ 23us) client_negotiation.cc:241] Negotiation successful
0429 14:03:39.759126 (+ 28us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260429 14:03:39.759873 3899 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:39.756871 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:37339 (local address 127.2.197.193:38115)
0429 14:03:39.756948 (+ 77us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:39.756950 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:39.756959 (+ 9us) server_negotiation.cc:408] Connection header received
0429 14:03:39.756985 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:39.756987 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:39.756995 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:39.757090 (+ 95us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:03:39.757465 (+ 375us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.758081 (+ 616us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:39.759191 (+ 1110us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.759737 (+ 546us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:39.759746 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:39.759771 (+ 25us) server_negotiation.cc:300] Negotiation successful
0429 14:03:39.759805 (+ 34us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260429 14:03:39.760087 3912 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:39.756943 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:45473 (local address 127.2.197.194:43639)
0429 14:03:39.757072 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:39.757083 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:39.757140 (+ 57us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:39.757308 (+ 168us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:39.757309 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:39.757311 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:03:39.757623 (+ 312us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:39.757625 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.758526 (+ 901us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:39.758527 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:39.759261 (+ 734us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:39.759263 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.759895 (+ 632us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:39.759899 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 14:03:39.759965 (+ 66us) client_negotiation.cc:241] Negotiation successful
0429 14:03:39.760008 (+ 43us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":69,"threads_started":1}
I20260429 14:03:39.760149 3902 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:39.757098 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:43639 (local address 127.2.197.195:45473)
0429 14:03:39.757144 (+ 46us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:39.757146 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:39.757178 (+ 32us) server_negotiation.cc:408] Connection header received
0429 14:03:39.757203 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:39.757204 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:39.757213 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:39.757265 (+ 52us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:03:39.757745 (+ 480us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.758419 (+ 674us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:39.759367 (+ 948us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.759855 (+ 488us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:39.759862 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:39.760020 (+ 158us) server_negotiation.cc:300] Negotiation successful
0429 14:03:39.760071 (+ 51us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260429 14:03:39.760174 3568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10738ffeb2314b858bb69ab5ec68fbd4" candidate_uuid: "35ad592ec35e473cbfe41566dad67848" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "474d8c15499f425f939fa1fd858d56c4" is_pre_election: true
I20260429 14:03:39.760347 3568 raft_consensus.cc:2468] T 10738ffeb2314b858bb69ab5ec68fbd4 P 474d8c15499f425f939fa1fd858d56c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35ad592ec35e473cbfe41566dad67848 in term 0.
I20260429 14:03:39.760428 3840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10738ffeb2314b858bb69ab5ec68fbd4" candidate_uuid: "35ad592ec35e473cbfe41566dad67848" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba1a455792e44812a83f24dd6f74464c" is_pre_election: true
I20260429 14:03:39.760532 3840 raft_consensus.cc:2468] T 10738ffeb2314b858bb69ab5ec68fbd4 P ba1a455792e44812a83f24dd6f74464c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35ad592ec35e473cbfe41566dad67848 in term 0.
I20260429 14:03:39.760689 3640 leader_election.cc:304] T 10738ffeb2314b858bb69ab5ec68fbd4 P 35ad592ec35e473cbfe41566dad67848 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35ad592ec35e473cbfe41566dad67848, 474d8c15499f425f939fa1fd858d56c4; no voters:
I20260429 14:03:39.760848 3908 raft_consensus.cc:2804] T 10738ffeb2314b858bb69ab5ec68fbd4 P 35ad592ec35e473cbfe41566dad67848 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 14:03:39.760931 3908 raft_consensus.cc:493] T 10738ffeb2314b858bb69ab5ec68fbd4 P 35ad592ec35e473cbfe41566dad67848 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 14:03:39.760967 3908 raft_consensus.cc:3060] T 10738ffeb2314b858bb69ab5ec68fbd4 P 35ad592ec35e473cbfe41566dad67848 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:03:39.761648 3908 raft_consensus.cc:515] T 10738ffeb2314b858bb69ab5ec68fbd4 P 35ad592ec35e473cbfe41566dad67848 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ad592ec35e473cbfe41566dad67848" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37585 } } peers { permanent_uuid: "474d8c15499f425f939fa1fd858d56c4" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 38115 } } peers { permanent_uuid: "ba1a455792e44812a83f24dd6f74464c" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 45473 } }
I20260429 14:03:39.761801 3908 leader_election.cc:290] T 10738ffeb2314b858bb69ab5ec68fbd4 P 35ad592ec35e473cbfe41566dad67848 [CANDIDATE]: Term 1 election: Requested vote from peers 474d8c15499f425f939fa1fd858d56c4 (127.2.197.193:38115), ba1a455792e44812a83f24dd6f74464c (127.2.197.195:45473)
I20260429 14:03:39.762008 3568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10738ffeb2314b858bb69ab5ec68fbd4" candidate_uuid: "35ad592ec35e473cbfe41566dad67848" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "474d8c15499f425f939fa1fd858d56c4"
I20260429 14:03:39.762106 3568 raft_consensus.cc:3060] T 10738ffeb2314b858bb69ab5ec68fbd4 P 474d8c15499f425f939fa1fd858d56c4 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:03:39.762074 3840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10738ffeb2314b858bb69ab5ec68fbd4" candidate_uuid: "35ad592ec35e473cbfe41566dad67848" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba1a455792e44812a83f24dd6f74464c"
I20260429 14:03:39.762171 3840 raft_consensus.cc:3060] T 10738ffeb2314b858bb69ab5ec68fbd4 P ba1a455792e44812a83f24dd6f74464c [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:03:39.762854 3568 raft_consensus.cc:2468] T 10738ffeb2314b858bb69ab5ec68fbd4 P 474d8c15499f425f939fa1fd858d56c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35ad592ec35e473cbfe41566dad67848 in term 1.
I20260429 14:03:39.762900 3840 raft_consensus.cc:2468] T 10738ffeb2314b858bb69ab5ec68fbd4 P ba1a455792e44812a83f24dd6f74464c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35ad592ec35e473cbfe41566dad67848 in term 1.
I20260429 14:03:39.763080 3640 leader_election.cc:304] T 10738ffeb2314b858bb69ab5ec68fbd4 P 35ad592ec35e473cbfe41566dad67848 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35ad592ec35e473cbfe41566dad67848, 474d8c15499f425f939fa1fd858d56c4; no voters:
I20260429 14:03:39.763275 3908 raft_consensus.cc:2804] T 10738ffeb2314b858bb69ab5ec68fbd4 P 35ad592ec35e473cbfe41566dad67848 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:03:39.763608 3908 raft_consensus.cc:697] T 10738ffeb2314b858bb69ab5ec68fbd4 P 35ad592ec35e473cbfe41566dad67848 [term 1 LEADER]: Becoming Leader. State: Replica: 35ad592ec35e473cbfe41566dad67848, State: Running, Role: LEADER
I20260429 14:03:39.763813 3908 consensus_queue.cc:237] T 10738ffeb2314b858bb69ab5ec68fbd4 P 35ad592ec35e473cbfe41566dad67848 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ad592ec35e473cbfe41566dad67848" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37585 } } peers { permanent_uuid: "474d8c15499f425f939fa1fd858d56c4" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 38115 } } peers { permanent_uuid: "ba1a455792e44812a83f24dd6f74464c" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 45473 } }
W20260429 14:03:39.764008 3751 tablet.cc:2404] T 10738ffeb2314b858bb69ab5ec68fbd4 P 35ad592ec35e473cbfe41566dad67848: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 14:03:39.764868 3420 catalog_manager.cc:5671] T 10738ffeb2314b858bb69ab5ec68fbd4 P 35ad592ec35e473cbfe41566dad67848 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35ad592ec35e473cbfe41566dad67848 (127.2.197.194). New cstate: current_term: 1 leader_uuid: "35ad592ec35e473cbfe41566dad67848" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ad592ec35e473cbfe41566dad67848" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "474d8c15499f425f939fa1fd858d56c4" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 38115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba1a455792e44812a83f24dd6f74464c" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 45473 } health_report { overall_health: UNKNOWN } } }
Apr 29 14:03:39 dist-test-slave-8n77 krb5kdc[3373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.193@KRBTEST.COM
I20260429 14:03:39.797175 3899 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:39.789453 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48964 (local address 127.2.197.193:38115)
0429 14:03:39.789639 (+ 186us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:39.789642 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:39.789804 (+ 162us) server_negotiation.cc:408] Connection header received
0429 14:03:39.789902 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:39.789903 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:39.789917 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:39.789994 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:39.790831 (+ 837us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.791275 (+ 444us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:39.791839 (+ 564us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.792027 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:39.794258 (+ 2231us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:39.794316 (+ 58us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:39.794317 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:39.794324 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:39.795895 (+ 1571us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:39.796251 (+ 356us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:39.796254 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:39.796254 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:39.796287 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:39.796475 (+ 188us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:39.796477 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:39.796477 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:39.796683 (+ 206us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:39.796898 (+ 215us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:39.796950 (+ 52us) server_negotiation.cc:300] Negotiation successful
0429 14:03:39.797055 (+ 105us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
Apr 29 14:03:39 dist-test-slave-8n77 krb5kdc[3373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471419, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Apr 29 14:03:39 dist-test-slave-8n77 krb5kdc[3373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471419, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.197.193@KRBTEST.COM
I20260429 14:03:39.820560 3899 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:39.812952 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48968 (local address 127.2.197.193:38115)
0429 14:03:39.813037 (+ 85us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:39.813039 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:39.813149 (+ 110us) server_negotiation.cc:408] Connection header received
0429 14:03:39.813260 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:39.813261 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:39.813270 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:39.813331 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:39.814097 (+ 766us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.814632 (+ 535us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:39.815216 (+ 584us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.815443 (+ 227us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:39.817630 (+ 2187us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:39.817640 (+ 10us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:39.817641 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:39.817649 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:39.819373 (+ 1724us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:39.819813 (+ 440us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:39.819814 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:39.819815 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:39.819847 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:39.819998 (+ 151us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:39.819999 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:39.820000 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:39.820090 (+ 90us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:39.820190 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:39.820348 (+ 158us) server_negotiation.cc:300] Negotiation successful
0429 14:03:39.820443 (+ 95us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Apr 29 14:03:39 dist-test-slave-8n77 krb5kdc[3373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Apr 29 14:03:39 dist-test-slave-8n77 krb5kdc[3373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.193@KRBTEST.COM
I20260429 14:03:39.845353 3899 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:39.837923 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48974 (local address 127.2.197.193:38115)
0429 14:03:39.837985 (+ 62us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:39.837988 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:39.838021 (+ 33us) server_negotiation.cc:408] Connection header received
0429 14:03:39.838095 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:39.838096 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:39.838105 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:39.838170 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:39.839146 (+ 976us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.839624 (+ 478us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:39.840292 (+ 668us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:39.840428 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:39.842789 (+ 2361us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:39.842795 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:39.842795 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:39.842803 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:39.844081 (+ 1278us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:39.844421 (+ 340us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:39.844422 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:39.844422 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:39.844445 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:39.844671 (+ 226us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:39.844672 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:39.844672 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:39.844782 (+ 110us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:39.844863 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:39.845124 (+ 261us) server_negotiation.cc:300] Negotiation successful
0429 14:03:39.845262 (+ 138us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260429 14:03:39.845911 3528 tablet_service.cc:3018] Rejecting scan request for tablet 10738ffeb2314b858bb69ab5ec68fbd4: Uninitialized: clean time has not yet been initialized
I20260429 14:03:39.847739 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 3485
I20260429 14:03:39.853193 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 3621
I20260429 14:03:39.858455 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 3757
I20260429 14:03:39.863682 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 3389
2026-04-29T14:03:39Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3348 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:03:39 dist-test-slave-8n77 krb5kdc[3927](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:03:39 dist-test-slave-8n77 krb5kdc[3927](info): set up 2 sockets
Apr 29 14:03:39 dist-test-slave-8n77 krb5kdc[3927](info): commencing operation
krb5kdc: starting...
W20260429 14:03:41.943672 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.057s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:03:42 dist-test-slave-8n77 krb5kdc[3927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:03:42Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:03:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:03:42.095089 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:38449
--webserver_interface=127.2.197.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:34441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:38449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:03:42.176819 3943 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:03:42.177027 3943 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:03:42.177052 3943 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:03:42.178836 3943 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:03:42.178898 3943 flags.cc:432] Enabled experimental flag: --txn_manager_enabled=true
W20260429 14:03:42.178912 3943 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:03:42.178926 3943 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:03:42.178938 3943 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:03:42.178951 3943 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:03:42.181712 3943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:34441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:38449
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:38449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:03:42.182075 3943 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:03:42.182331 3943 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:03:42.185665 3949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:03:42.185657 3948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:03:42.185657 3951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:03:42.185950 3943 server_base.cc:1061] running on GCE node
I20260429 14:03:42.186260 3943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:03:42.186578 3943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:03:42.187731 3943 hybrid_clock.cc:648] HybridClock initialized: now 1777471422187687 us; error 60 us; skew 500 ppm
Apr 29 14:03:42 dist-test-slave-8n77 krb5kdc[3927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471422, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:03:42.190624 3943 init.cc:377] Logged in from keytab as kudu/127.2.197.254@KRBTEST.COM (short username kudu)
I20260429 14:03:42.191206 3943 webserver.cc:492] Webserver started at http://127.2.197.254:40977/ using document root <none> and password file <none>
I20260429 14:03:42.191406 3943 fs_manager.cc:362] Metadata directory not provided
I20260429 14:03:42.191466 3943 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:03:42.191570 3943 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:03:42.192548 3943 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "5fc55c28f11d408ebaf8a4aac4312b83"
format_stamp: "Formatted at 2026-04-29 14:03:42 on dist-test-slave-8n77"
server_key: "192312320edf34a775286e9b23208888"
server_key_iv: "062b15c9c5d57adb9dae6669b7c6b7e8"
server_key_version: "encryptionkey@0"
I20260429 14:03:42.192885 3943 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "5fc55c28f11d408ebaf8a4aac4312b83"
format_stamp: "Formatted at 2026-04-29 14:03:42 on dist-test-slave-8n77"
server_key: "192312320edf34a775286e9b23208888"
server_key_iv: "062b15c9c5d57adb9dae6669b7c6b7e8"
server_key_version: "encryptionkey@0"
I20260429 14:03:42.194183 3943 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260429 14:03:42.195116 3958 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:42.195689 3943 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:03:42.195837 3943 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "5fc55c28f11d408ebaf8a4aac4312b83"
format_stamp: "Formatted at 2026-04-29 14:03:42 on dist-test-slave-8n77"
server_key: "192312320edf34a775286e9b23208888"
server_key_iv: "062b15c9c5d57adb9dae6669b7c6b7e8"
server_key_version: "encryptionkey@0"
I20260429 14:03:42.195942 3943 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:03:42.212491 3943 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:03:42.220731 3943 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:03:42.220963 3943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:03:42.227265 3943 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:38449
I20260429 14:03:42.227316 4020 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:38449 every 8 connection(s)
I20260429 14:03:42.227766 3943 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:03:42.228776 4021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:03:42.230574 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 3943
I20260429 14:03:42.230674 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:03:42.230832 2839 external_mini_cluster.cc:1468] Setting key 3309381824f51e8d5f0244b1090aa2a2
I20260429 14:03:42.232120 4021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fc55c28f11d408ebaf8a4aac4312b83: Bootstrap starting.
I20260429 14:03:42.233196 4021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fc55c28f11d408ebaf8a4aac4312b83: Neither blocks nor log segments found. Creating new log.
I20260429 14:03:42.233518 4021 log.cc:826] T 00000000000000000000000000000000 P 5fc55c28f11d408ebaf8a4aac4312b83: Log is configured to *not* fsync() on all Append() calls
I20260429 14:03:42.234361 4021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fc55c28f11d408ebaf8a4aac4312b83: No bootstrap required, opened a new log
I20260429 14:03:42.236129 4021 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5fc55c28f11d408ebaf8a4aac4312b83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc55c28f11d408ebaf8a4aac4312b83" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 38449 } }
I20260429 14:03:42.236320 4021 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5fc55c28f11d408ebaf8a4aac4312b83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:03:42.236351 4021 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5fc55c28f11d408ebaf8a4aac4312b83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fc55c28f11d408ebaf8a4aac4312b83, State: Initialized, Role: FOLLOWER
I20260429 14:03:42.236469 4021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fc55c28f11d408ebaf8a4aac4312b83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc55c28f11d408ebaf8a4aac4312b83" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 38449 } }
I20260429 14:03:42.236550 4021 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5fc55c28f11d408ebaf8a4aac4312b83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:03:42.236604 4021 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5fc55c28f11d408ebaf8a4aac4312b83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:03:42.236661 4021 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5fc55c28f11d408ebaf8a4aac4312b83 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:03:42.237519 4021 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5fc55c28f11d408ebaf8a4aac4312b83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc55c28f11d408ebaf8a4aac4312b83" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 38449 } }
I20260429 14:03:42.237627 4021 leader_election.cc:304] T 00000000000000000000000000000000 P 5fc55c28f11d408ebaf8a4aac4312b83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fc55c28f11d408ebaf8a4aac4312b83; no voters:
Apr 29 14:03:42 dist-test-slave-8n77 krb5kdc[3927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:03:42.237846 4021 leader_election.cc:290] T 00000000000000000000000000000000 P 5fc55c28f11d408ebaf8a4aac4312b83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:03:42.238018 4026 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5fc55c28f11d408ebaf8a4aac4312b83 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:03:42.238108 4021 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5fc55c28f11d408ebaf8a4aac4312b83 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:03:42.238132 4026 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5fc55c28f11d408ebaf8a4aac4312b83 [term 1 LEADER]: Becoming Leader. State: Replica: 5fc55c28f11d408ebaf8a4aac4312b83, State: Running, Role: LEADER
I20260429 14:03:42.238204 4026 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fc55c28f11d408ebaf8a4aac4312b83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc55c28f11d408ebaf8a4aac4312b83" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 38449 } }
I20260429 14:03:42.238564 4027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fc55c28f11d408ebaf8a4aac4312b83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fc55c28f11d408ebaf8a4aac4312b83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc55c28f11d408ebaf8a4aac4312b83" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 38449 } } }
I20260429 14:03:42.238545 4026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fc55c28f11d408ebaf8a4aac4312b83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fc55c28f11d408ebaf8a4aac4312b83. Latest consensus state: current_term: 1 leader_uuid: "5fc55c28f11d408ebaf8a4aac4312b83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc55c28f11d408ebaf8a4aac4312b83" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 38449 } } }
I20260429 14:03:42.238724 4027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fc55c28f11d408ebaf8a4aac4312b83 [sys.catalog]: This master's current role is: LEADER
I20260429 14:03:42.239058 4026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fc55c28f11d408ebaf8a4aac4312b83 [sys.catalog]: This master's current role is: LEADER
I20260429 14:03:42.239398 4035 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:03:42.239966 4035 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:03:42.241511 4035 catalog_manager.cc:1357] Generated new cluster ID: 214368d225a94aeaa2b14f49108842bc
I20260429 14:03:42.241562 4035 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:03:42.242478 4024 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:42.231690 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49798 (local address 127.2.197.254:38449)
0429 14:03:42.231919 (+ 229us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:42.231931 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:42.231957 (+ 26us) server_negotiation.cc:408] Connection header received
0429 14:03:42.232437 (+ 480us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:42.232445 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:42.232499 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:42.232648 (+ 149us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:42.233773 (+ 1125us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.234756 (+ 983us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:42.235646 (+ 890us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.235903 (+ 257us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:42.238470 (+ 2567us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:42.238479 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:42.238480 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:42.238495 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:42.240683 (+ 2188us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:42.241099 (+ 416us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:42.241102 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:42.241102 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:42.241153 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:42.241364 (+ 211us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:42.241365 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:42.241365 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:42.241613 (+ 248us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:42.242113 (+ 500us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:42.242184 (+ 71us) server_negotiation.cc:300] Negotiation successful
0429 14:03:42.242325 (+ 141us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":139,"threads_started":1}
I20260429 14:03:42.257336 4035 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:03:42.257921 4035 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:03:42.268821 4035 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 5fc55c28f11d408ebaf8a4aac4312b83: Generated new TSK 0
I20260429 14:03:42.269048 4035 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:03:42.330209 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.193:0
--local_ip_for_outbound_sockets=127.2.197.193
--webserver_interface=127.2.197.193
--webserver_port=0
--tserver_master_addrs=127.2.197.254:38449
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:34441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:03:42.424247 4049 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:03:42.424427 4049 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:03:42.424453 4049 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:03:42.426119 4049 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:03:42.426172 4049 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:03:42.426194 4049 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260429 14:03:42.426211 4049 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.193
I20260429 14:03:42.427956 4049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:34441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.197.193
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.197.254:38449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.193
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:03:42.428184 4049 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:03:42.428385 4049 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:03:42.430913 4057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:03:42.430924 4054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:03:42.430915 4055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:03:42.431357 4049 server_base.cc:1061] running on GCE node
I20260429 14:03:42.431535 4049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:03:42.431754 4049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:03:42.432934 4049 hybrid_clock.cc:648] HybridClock initialized: now 1777471422432904 us; error 44 us; skew 500 ppm
Apr 29 14:03:42 dist-test-slave-8n77 krb5kdc[3927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471422, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:03:42.435498 4049 init.cc:377] Logged in from keytab as kudu/127.2.197.193@KRBTEST.COM (short username kudu)
I20260429 14:03:42.436041 4049 webserver.cc:492] Webserver started at http://127.2.197.193:44273/ using document root <none> and password file <none>
I20260429 14:03:42.436190 4049 fs_manager.cc:362] Metadata directory not provided
I20260429 14:03:42.436225 4049 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:03:42.436295 4049 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:03:42.437157 4049 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-0/data/instance:
uuid: "6a2f154bcc534134bd963d22d54ba278"
format_stamp: "Formatted at 2026-04-29 14:03:42 on dist-test-slave-8n77"
server_key: "646298beb52eb1c8367a525742565c68"
server_key_iv: "09b2086546d5e44debde7a1e45daf6d9"
server_key_version: "encryptionkey@0"
I20260429 14:03:42.437423 4049 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance:
uuid: "6a2f154bcc534134bd963d22d54ba278"
format_stamp: "Formatted at 2026-04-29 14:03:42 on dist-test-slave-8n77"
server_key: "646298beb52eb1c8367a525742565c68"
server_key_iv: "09b2086546d5e44debde7a1e45daf6d9"
server_key_version: "encryptionkey@0"
I20260429 14:03:42.438719 4049 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260429 14:03:42.439685 4064 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:42.439924 4049 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:03:42.440016 4049 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-0/wal
uuid: "6a2f154bcc534134bd963d22d54ba278"
format_stamp: "Formatted at 2026-04-29 14:03:42 on dist-test-slave-8n77"
server_key: "646298beb52eb1c8367a525742565c68"
server_key_iv: "09b2086546d5e44debde7a1e45daf6d9"
server_key_version: "encryptionkey@0"
I20260429 14:03:42.440083 4049 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:03:42.462092 4049 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:03:42.464565 4049 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:03:42.464740 4049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:03:42.465299 4049 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:03:42.465342 4049 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:42.465366 4049 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:03:42.465379 4049 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:42.472443 4049 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.193:41417
I20260429 14:03:42.472739 4179 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.193:41417 every 8 connection(s)
I20260429 14:03:42.472882 4049 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
Apr 29 14:03:42 dist-test-slave-8n77 krb5kdc[3927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471422, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:03:42.476351 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 4049
I20260429 14:03:42.476461 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance
I20260429 14:03:42.476663 2839 external_mini_cluster.cc:1468] Setting key 4e48b2949f049be21c50787d687c7642
I20260429 14:03:42.477514 4073 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:42.466191 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:38449 (local address 127.2.197.193:47331)
0429 14:03:42.466455 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:42.466487 (+ 32us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:42.467286 (+ 799us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:42.467747 (+ 461us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:42.467753 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:42.468239 (+ 486us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:03:42.468669 (+ 430us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:42.468674 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.469809 (+ 1135us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:42.469811 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:42.470504 (+ 693us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:42.470508 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.470623 (+ 115us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:42.471176 (+ 553us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:03:42.471191 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:03:42.473883 (+ 2692us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:03:42.475941 (+ 2058us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:42.475947 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:42.475949 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:42.476180 (+ 231us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:42.476424 (+ 244us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:42.476425 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:42.476425 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:42.476503 (+ 78us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:42.476867 (+ 364us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:03:42.476872 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:03:42.477057 (+ 185us) client_negotiation.cc:770] Sending connection context
0429 14:03:42.477177 (+ 120us) client_negotiation.cc:241] Negotiation successful
0429 14:03:42.477337 (+ 160us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":191,"threads_started":1}
I20260429 14:03:42.477655 4024 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:42.466198 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:47331 (local address 127.2.197.254:38449)
0429 14:03:42.466306 (+ 108us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:42.466308 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:42.466594 (+ 286us) server_negotiation.cc:408] Connection header received
0429 14:03:42.467432 (+ 838us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:42.467434 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:42.467450 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:42.467540 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:42.468818 (+ 1278us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.469693 (+ 875us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:42.470614 (+ 921us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.470825 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:42.474008 (+ 3183us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:42.474014 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:42.474015 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:42.474024 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:42.475837 (+ 1813us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:42.476287 (+ 450us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:42.476289 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:42.476289 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:42.476325 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:42.476583 (+ 258us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:42.476584 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:42.476585 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:42.476733 (+ 148us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:42.476826 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:42.477394 (+ 568us) server_negotiation.cc:300] Negotiation successful
0429 14:03:42.477503 (+ 109us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260429 14:03:42.478765 4180 heartbeater.cc:344] Connected to a master server at 127.2.197.254:38449
I20260429 14:03:42.478881 4180 heartbeater.cc:461] Registering TS with master...
I20260429 14:03:42.479090 4180 heartbeater.cc:507] Master 127.2.197.254:38449 requested a full tablet report, sending...
I20260429 14:03:42.479619 3975 ts_manager.cc:194] Registered new tserver with Master: 6a2f154bcc534134bd963d22d54ba278 (127.2.197.193:41417)
I20260429 14:03:42.480540 3975 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.193@KRBTEST.COM'} at 127.2.197.193:47331
I20260429 14:03:42.487102 4024 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:42.480085 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:54009 (local address 127.2.197.254:38449)
0429 14:03:42.480130 (+ 45us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:42.480132 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:42.480231 (+ 99us) server_negotiation.cc:408] Connection header received
0429 14:03:42.480281 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:42.480282 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:42.480291 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:42.480339 (+ 48us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:42.480852 (+ 513us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.481571 (+ 719us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:42.482528 (+ 957us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.482721 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:42.483702 (+ 981us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:42.483709 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:42.483709 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:42.483717 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:42.485509 (+ 1792us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:42.485974 (+ 465us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:42.485977 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:42.485977 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:42.486013 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:42.486242 (+ 229us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:42.486243 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:42.486243 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:42.486439 (+ 196us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:42.486522 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:42.486924 (+ 402us) server_negotiation.cc:300] Negotiation successful
0429 14:03:42.487030 (+ 106us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260429 14:03:42.487195 4187 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:42.479992 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:38449 (local address 127.2.197.193:54009)
0429 14:03:42.480150 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:42.480162 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:42.480226 (+ 64us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:42.480344 (+ 118us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:42.480345 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:42.480519 (+ 174us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:03:42.480748 (+ 229us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:42.480751 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.481668 (+ 917us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:42.481669 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:42.482345 (+ 676us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:42.482349 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.482431 (+ 82us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:42.483161 (+ 730us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:03:42.483167 (+ 6us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:03:42.483572 (+ 405us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:03:42.485628 (+ 2056us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:42.485629 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:42.485630 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:42.485844 (+ 214us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:42.486099 (+ 255us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:42.486100 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:42.486101 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:42.486139 (+ 38us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:42.486613 (+ 474us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:03:42.486615 (+ 2us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:03:42.486763 (+ 148us) client_negotiation.cc:770] Sending connection context
0429 14:03:42.486882 (+ 119us) client_negotiation.cc:241] Negotiation successful
0429 14:03:42.487054 (+ 172us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for kudu/127.2.197.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:03:42.538223 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.194:0
--local_ip_for_outbound_sockets=127.2.197.194
--webserver_interface=127.2.197.194
--webserver_port=0
--tserver_master_addrs=127.2.197.254:38449
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:34441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:03:42.618753 4191 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:03:42.618970 4191 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:03:42.619014 4191 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:03:42.621865 4191 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:03:42.621975 4191 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:03:42.622017 4191 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260429 14:03:42.622061 4191 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.194
I20260429 14:03:42.624693 4191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:34441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.197.194
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.197.254:38449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.194
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:03:42.625034 4191 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:03:42.625237 4191 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:03:42.629765 4197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:03:42.629951 4199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:03:42.629858 4196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:03:42.629943 4191 server_base.cc:1061] running on GCE node
I20260429 14:03:42.630487 4191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:03:42.630784 4191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:03:42.631956 4191 hybrid_clock.cc:648] HybridClock initialized: now 1777471422631929 us; error 40 us; skew 500 ppm
Apr 29 14:03:42 dist-test-slave-8n77 krb5kdc[3927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471422, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:03:42.635432 4191 init.cc:377] Logged in from keytab as kudu/127.2.197.194@KRBTEST.COM (short username kudu)
I20260429 14:03:42.636077 4191 webserver.cc:492] Webserver started at http://127.2.197.194:36919/ using document root <none> and password file <none>
I20260429 14:03:42.636282 4191 fs_manager.cc:362] Metadata directory not provided
I20260429 14:03:42.636343 4191 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:03:42.636448 4191 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:03:42.637415 4191 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-1/data/instance:
uuid: "7de3fbd2cbcf477fa2b3dfc27674e490"
format_stamp: "Formatted at 2026-04-29 14:03:42 on dist-test-slave-8n77"
server_key: "5255b9ef27352003bde3f7ed8705474e"
server_key_iv: "cac69158dca1b0b4bc84e6b0eb5d7975"
server_key_version: "encryptionkey@0"
I20260429 14:03:42.637750 4191 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance:
uuid: "7de3fbd2cbcf477fa2b3dfc27674e490"
format_stamp: "Formatted at 2026-04-29 14:03:42 on dist-test-slave-8n77"
server_key: "5255b9ef27352003bde3f7ed8705474e"
server_key_iv: "cac69158dca1b0b4bc84e6b0eb5d7975"
server_key_version: "encryptionkey@0"
I20260429 14:03:42.639235 4191 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260429 14:03:42.640249 4206 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:42.640475 4191 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:03:42.640563 4191 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-1/wal
uuid: "7de3fbd2cbcf477fa2b3dfc27674e490"
format_stamp: "Formatted at 2026-04-29 14:03:42 on dist-test-slave-8n77"
server_key: "5255b9ef27352003bde3f7ed8705474e"
server_key_iv: "cac69158dca1b0b4bc84e6b0eb5d7975"
server_key_version: "encryptionkey@0"
I20260429 14:03:42.640843 4191 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:03:42.650676 4191 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:03:42.652684 4191 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:03:42.652853 4191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:03:42.653501 4191 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:03:42.653551 4191 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:42.653597 4191 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:03:42.653618 4191 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 29 14:03:42 dist-test-slave-8n77 krb5kdc[3927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471422, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:03:42.661099 4191 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.194:40607
I20260429 14:03:42.661233 4321 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.194:40607 every 8 connection(s)
I20260429 14:03:42.661489 4191 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
I20260429 14:03:42.663352 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 4191
I20260429 14:03:42.663497 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance
I20260429 14:03:42.663697 2839 external_mini_cluster.cc:1468] Setting key 787f93c50d1f0a2997c9ddc7ad2f6d64
I20260429 14:03:42.666054 4024 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:42.654231 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:60931 (local address 127.2.197.254:38449)
0429 14:03:42.654302 (+ 71us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:42.654305 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:42.654317 (+ 12us) server_negotiation.cc:408] Connection header received
0429 14:03:42.654980 (+ 663us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:42.654981 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:42.654997 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:42.655079 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:42.656246 (+ 1167us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.657115 (+ 869us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:42.658032 (+ 917us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.658298 (+ 266us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:42.661781 (+ 3483us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:42.661788 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:42.661789 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:42.661799 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:42.663710 (+ 1911us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:42.664297 (+ 587us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:42.664299 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:42.664299 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:42.664327 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:42.664661 (+ 334us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:42.664662 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:42.664662 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:42.664808 (+ 146us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:42.664903 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:42.665584 (+ 681us) server_negotiation.cc:300] Negotiation successful
0429 14:03:42.665893 (+ 309us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260429 14:03:42.666158 4215 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:42.653886 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:38449 (local address 127.2.197.194:60931)
0429 14:03:42.654063 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:42.654104 (+ 41us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:42.654827 (+ 723us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:42.655160 (+ 333us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:42.655164 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:42.655665 (+ 501us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:03:42.656104 (+ 439us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:42.656110 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.657240 (+ 1130us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:42.657241 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:42.657931 (+ 690us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:42.657936 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.658084 (+ 148us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:42.658813 (+ 729us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:03:42.658826 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:03:42.661641 (+ 2815us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:03:42.663836 (+ 2195us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:42.663846 (+ 10us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:42.663850 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:42.664202 (+ 352us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:42.664399 (+ 197us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:42.664400 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:42.664401 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:42.664572 (+ 171us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:42.664890 (+ 318us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:03:42.664896 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:03:42.665263 (+ 367us) client_negotiation.cc:770] Sending connection context
0429 14:03:42.665529 (+ 266us) client_negotiation.cc:241] Negotiation successful
0429 14:03:42.665897 (+ 368us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":92,"threads_started":1}
I20260429 14:03:42.667137 4322 heartbeater.cc:344] Connected to a master server at 127.2.197.254:38449
I20260429 14:03:42.667253 4322 heartbeater.cc:461] Registering TS with master...
I20260429 14:03:42.667593 4322 heartbeater.cc:507] Master 127.2.197.254:38449 requested a full tablet report, sending...
I20260429 14:03:42.668192 3974 ts_manager.cc:194] Registered new tserver with Master: 7de3fbd2cbcf477fa2b3dfc27674e490 (127.2.197.194:40607)
I20260429 14:03:42.668641 3974 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.194@KRBTEST.COM'} at 127.2.197.194:60931
I20260429 14:03:42.676666 4329 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:42.670714 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:38449 (local address 127.2.197.194:40681)
0429 14:03:42.670980 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:42.670992 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:42.671083 (+ 91us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:42.671268 (+ 185us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:42.671269 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:42.671443 (+ 174us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:03:42.671625 (+ 182us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:42.671627 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.672285 (+ 658us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:42.672286 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:42.672667 (+ 381us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:42.672669 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.672752 (+ 83us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:42.673352 (+ 600us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:03:42.673356 (+ 4us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:03:42.673658 (+ 302us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:03:42.675603 (+ 1945us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:42.675604 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:42.675605 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:42.675768 (+ 163us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:42.675968 (+ 200us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:42.675969 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:42.675970 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:42.676007 (+ 37us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:42.676274 (+ 267us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:03:42.676275 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:03:42.676341 (+ 66us) client_negotiation.cc:770] Sending connection context
0429 14:03:42.676418 (+ 77us) client_negotiation.cc:241] Negotiation successful
0429 14:03:42.676538 (+ 120us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":105,"threads_started":1}
I20260429 14:03:42.676653 4024 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:42.670761 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:40681 (local address 127.2.197.254:38449)
0429 14:03:42.670823 (+ 62us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:42.670825 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:42.671066 (+ 241us) server_negotiation.cc:408] Connection header received
0429 14:03:42.671169 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:42.671171 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:42.671181 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:42.671261 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:42.671709 (+ 448us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.672194 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:42.672749 (+ 555us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.672960 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:42.673763 (+ 803us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:42.673767 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:42.673767 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:42.673774 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:42.675440 (+ 1666us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:42.675874 (+ 434us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:42.675875 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:42.675875 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:42.675903 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:42.676078 (+ 175us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:42.676078 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:42.676079 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:42.676207 (+ 128us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:42.676285 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:42.676445 (+ 160us) server_negotiation.cc:300] Negotiation successful
0429 14:03:42.676549 (+ 104us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
WARNING: no policy specified for kudu/127.2.197.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:03:42.734208 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.195:0
--local_ip_for_outbound_sockets=127.2.197.195
--webserver_interface=127.2.197.195
--webserver_port=0
--tserver_master_addrs=127.2.197.254:38449
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:34441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:03:42.813393 4333 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:03:42.813586 4333 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:03:42.813613 4333 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:03:42.815130 4333 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:03:42.815183 4333 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:03:42.815205 4333 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260429 14:03:42.815223 4333 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.195
I20260429 14:03:42.816876 4333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:34441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.197.195
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.197.254:38449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.195
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:03:42.817080 4333 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:03:42.817221 4333 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:03:42.820504 4339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:03:42.820580 4333 server_base.cc:1061] running on GCE node
W20260429 14:03:42.820451 4338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:03:42.820451 4341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:03:42.820956 4333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:03:42.821159 4333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:03:42.822341 4333 hybrid_clock.cc:648] HybridClock initialized: now 1777471422822305 us; error 50 us; skew 500 ppm
Apr 29 14:03:42 dist-test-slave-8n77 krb5kdc[3927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471422, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:03:42.824993 4333 init.cc:377] Logged in from keytab as kudu/127.2.197.195@KRBTEST.COM (short username kudu)
I20260429 14:03:42.825531 4333 webserver.cc:492] Webserver started at http://127.2.197.195:34905/ using document root <none> and password file <none>
I20260429 14:03:42.825759 4333 fs_manager.cc:362] Metadata directory not provided
I20260429 14:03:42.825801 4333 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:03:42.825912 4333 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:03:42.826896 4333 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-2/data/instance:
uuid: "52af864dbc164356a9ef9b9974c6be29"
format_stamp: "Formatted at 2026-04-29 14:03:42 on dist-test-slave-8n77"
server_key: "2601b0c6b72c4fe989287b7255bd77d9"
server_key_iv: "69f1f41f40b95e98d2ac360708665621"
server_key_version: "encryptionkey@0"
I20260429 14:03:42.827209 4333 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance:
uuid: "52af864dbc164356a9ef9b9974c6be29"
format_stamp: "Formatted at 2026-04-29 14:03:42 on dist-test-slave-8n77"
server_key: "2601b0c6b72c4fe989287b7255bd77d9"
server_key_iv: "69f1f41f40b95e98d2ac360708665621"
server_key_version: "encryptionkey@0"
I20260429 14:03:42.828431 4333 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260429 14:03:42.829185 4348 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:42.829401 4333 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:03:42.829501 4333 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-2/wal
uuid: "52af864dbc164356a9ef9b9974c6be29"
format_stamp: "Formatted at 2026-04-29 14:03:42 on dist-test-slave-8n77"
server_key: "2601b0c6b72c4fe989287b7255bd77d9"
server_key_iv: "69f1f41f40b95e98d2ac360708665621"
server_key_version: "encryptionkey@0"
I20260429 14:03:42.829597 4333 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:03:42.842816 4333 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:03:42.845137 4333 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:03:42.845343 4333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:03:42.846012 4333 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:03:42.846071 4333 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:42.846108 4333 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:03:42.846136 4333 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:42.853852 4333 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.195:33361
I20260429 14:03:42.853916 4463 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.195:33361 every 8 connection(s)
Apr 29 14:03:42 dist-test-slave-8n77 krb5kdc[3927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471422, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:03:42.854458 4333 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
I20260429 14:03:42.858546 4024 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:42.846571 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:44101 (local address 127.2.197.254:38449)
0429 14:03:42.846666 (+ 95us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:42.846670 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:42.847209 (+ 539us) server_negotiation.cc:408] Connection header received
0429 14:03:42.848078 (+ 869us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:42.848080 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:42.848095 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:42.848141 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:42.849222 (+ 1081us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.850080 (+ 858us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:42.850966 (+ 886us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.851189 (+ 223us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:42.855092 (+ 3903us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:42.855100 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:42.855101 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:42.855112 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:42.856794 (+ 1682us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:42.857297 (+ 503us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:42.857299 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:42.857299 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:42.857325 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:42.857604 (+ 279us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:42.857606 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:42.857607 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:42.857769 (+ 162us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:42.857864 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:42.858254 (+ 390us) server_negotiation.cc:300] Negotiation successful
0429 14:03:42.858435 (+ 181us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260429 14:03:42.858610 4357 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:42.846621 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:38449 (local address 127.2.197.195:44101)
0429 14:03:42.847026 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:42.847070 (+ 44us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:42.847906 (+ 836us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:42.848207 (+ 301us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:42.848212 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:42.848648 (+ 436us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:03:42.849101 (+ 453us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:42.849105 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.850186 (+ 1081us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:42.850187 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:42.850862 (+ 675us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:42.850864 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.851001 (+ 137us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:42.851607 (+ 606us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:03:42.851618 (+ 11us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:03:42.854914 (+ 3296us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:03:42.856921 (+ 2007us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:42.856928 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:42.856930 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:42.857192 (+ 262us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:42.857397 (+ 205us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:42.857397 (+ 0us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:42.857398 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:42.857456 (+ 58us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:42.857888 (+ 432us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:03:42.857893 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:03:42.858036 (+ 143us) client_negotiation.cc:770] Sending connection context
0429 14:03:42.858159 (+ 123us) client_negotiation.cc:241] Negotiation successful
0429 14:03:42.858349 (+ 190us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":314,"threads_started":1}
I20260429 14:03:42.859153 4464 heartbeater.cc:344] Connected to a master server at 127.2.197.254:38449
I20260429 14:03:42.859265 4464 heartbeater.cc:461] Registering TS with master...
I20260429 14:03:42.859391 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 4333
I20260429 14:03:42.859481 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance
I20260429 14:03:42.859493 4464 heartbeater.cc:507] Master 127.2.197.254:38449 requested a full tablet report, sending...
I20260429 14:03:42.859663 2839 external_mini_cluster.cc:1468] Setting key 0c2b9aec9d0665c3a30251587f975df3
I20260429 14:03:42.859951 3975 ts_manager.cc:194] Registered new tserver with Master: 52af864dbc164356a9ef9b9974c6be29 (127.2.197.195:33361)
I20260429 14:03:42.860456 3975 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.195@KRBTEST.COM'} at 127.2.197.195:44101
I20260429 14:03:42.861248 2839 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260429 14:03:42.867524 4024 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:42.860431 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:39443 (local address 127.2.197.254:38449)
0429 14:03:42.860487 (+ 56us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:42.860489 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:42.860566 (+ 77us) server_negotiation.cc:408] Connection header received
0429 14:03:42.860634 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:42.860635 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:42.860646 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:42.860695 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:42.861252 (+ 557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.861858 (+ 606us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:42.862562 (+ 704us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.862754 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:42.863739 (+ 985us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:42.863747 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:42.863747 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:42.863757 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:42.865714 (+ 1957us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:42.866098 (+ 384us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:42.866099 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:42.866099 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:42.866362 (+ 263us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:42.866912 (+ 550us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:42.866915 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:42.866915 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:42.867076 (+ 161us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:42.867179 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:42.867358 (+ 179us) server_negotiation.cc:300] Negotiation successful
0429 14:03:42.867451 (+ 93us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260429 14:03:42.867697 4470 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:42.860352 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:38449 (local address 127.2.197.195:39443)
0429 14:03:42.860489 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:42.860501 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:42.860550 (+ 49us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:42.860717 (+ 167us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:42.860718 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:42.860924 (+ 206us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:03:42.861184 (+ 260us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:42.861187 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.861998 (+ 811us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:42.861999 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:42.862460 (+ 461us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:42.862462 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.862564 (+ 102us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:42.863150 (+ 586us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:03:42.863154 (+ 4us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:03:42.863536 (+ 382us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:03:42.865840 (+ 2304us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:42.865843 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:42.865843 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:42.866016 (+ 173us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:42.866538 (+ 522us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:42.866541 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:42.866541 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:42.866611 (+ 70us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:42.867174 (+ 563us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:03:42.867176 (+ 2us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:03:42.867225 (+ 49us) client_negotiation.cc:770] Sending connection context
0429 14:03:42.867294 (+ 69us) client_negotiation.cc:241] Negotiation successful
0429 14:03:42.867486 (+ 192us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":77,"threads_started":1}
Apr 29 14:03:42 dist-test-slave-8n77 krb5kdc[3927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471422, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Apr 29 14:03:42 dist-test-slave-8n77 krb5kdc[3927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471422, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:03:42.886499 4024 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:42.878973 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49802 (local address 127.2.197.254:38449)
0429 14:03:42.879064 (+ 91us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:42.879067 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:42.879077 (+ 10us) server_negotiation.cc:408] Connection header received
0429 14:03:42.879109 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:42.879111 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:42.879125 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:42.879206 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:42.879993 (+ 787us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.880493 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:42.881162 (+ 669us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.881347 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:42.883537 (+ 2190us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:42.883543 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:42.883543 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:42.883552 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:42.885066 (+ 1514us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:42.885432 (+ 366us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:42.885432 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:42.885433 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:42.885453 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:42.885849 (+ 396us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:42.885851 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:42.885851 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:42.885957 (+ 106us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:42.886062 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:42.886334 (+ 272us) server_negotiation.cc:300] Negotiation successful
0429 14:03:42.886427 (+ 93us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260429 14:03:42.888432 3974 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 14:03:42.889756 3974 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 14:03:42.896631 4482 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:42.892673 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:33361 (local address 127.0.0.1:48846)
0429 14:03:42.892922 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:42.892988 (+ 66us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:42.893094 (+ 106us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:42.893880 (+ 786us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:42.893884 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:42.893886 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:03:42.894196 (+ 310us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:42.894199 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.895547 (+ 1348us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:42.895548 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:42.896291 (+ 743us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:42.896293 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.896387 (+ 94us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:42.896431 (+ 44us) client_negotiation.cc:770] Sending connection context
0429 14:03:42.896483 (+ 52us) client_negotiation.cc:241] Negotiation successful
0429 14:03:42.896529 (+ 46us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":73,"threads_started":1}
I20260429 14:03:42.897039 4483 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:42.892710 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:41417 (local address 127.0.0.1:42718)
0429 14:03:42.893074 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:42.893087 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:42.893242 (+ 155us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:42.894219 (+ 977us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:42.894220 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:42.894222 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:03:42.894475 (+ 253us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:42.894477 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.895886 (+ 1409us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:42.895888 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:42.896814 (+ 926us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:42.896817 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.896887 (+ 70us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:42.896890 (+ 3us) client_negotiation.cc:770] Sending connection context
0429 14:03:42.896923 (+ 33us) client_negotiation.cc:241] Negotiation successful
0429 14:03:42.896968 (+ 45us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":153,"threads_started":1}
I20260429 14:03:42.897142 4484 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:42.893044 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48846 (local address 127.2.197.195:33361)
0429 14:03:42.893638 (+ 594us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:42.893648 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:42.893658 (+ 10us) server_negotiation.cc:408] Connection header received
0429 14:03:42.893720 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:42.893722 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:42.893740 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:42.893813 (+ 73us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:03:42.894314 (+ 501us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.895453 (+ 1139us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:42.896407 (+ 954us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.896915 (+ 508us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:42.896970 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:42.897002 (+ 32us) server_negotiation.cc:300] Negotiation successful
0429 14:03:42.897050 (+ 48us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":90,"threads_started":1}
I20260429 14:03:42.897686 4398 tablet_service.cc:1511] Processing CreateTablet for tablet 010d9785e99148d78bc921ce83beff6a (DEFAULT_TABLE table=test-table [id=38cf6dc750fa4de2a0774be0ce5d9e9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:03:42.898003 4398 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 010d9785e99148d78bc921ce83beff6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:03:42.898255 4481 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:42.892899 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:40607 (local address 127.0.0.1:37116)
0429 14:03:42.893798 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:42.893809 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:42.893859 (+ 50us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:42.895464 (+ 1605us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:42.895465 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:42.895467 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:03:42.895679 (+ 212us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:42.895680 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.897014 (+ 1334us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:42.897015 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:42.898056 (+ 1041us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:42.898059 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.898124 (+ 65us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:42.898127 (+ 3us) client_negotiation.cc:770] Sending connection context
0429 14:03:42.898157 (+ 30us) client_negotiation.cc:241] Negotiation successful
0429 14:03:42.898197 (+ 40us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":884,"thread_start_us":70,"threads_started":1}
I20260429 14:03:42.898906 4485 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:42.893209 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42718 (local address 127.2.197.193:41417)
0429 14:03:42.893979 (+ 770us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:42.893986 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:42.893995 (+ 9us) server_negotiation.cc:408] Connection header received
0429 14:03:42.894033 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:42.894035 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:42.894049 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:42.894115 (+ 66us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:03:42.894563 (+ 448us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.895784 (+ 1221us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:42.897212 (+ 1428us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.898735 (+ 1523us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:42.898771 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:42.898799 (+ 28us) server_negotiation.cc:300] Negotiation successful
0429 14:03:42.898850 (+ 51us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":76,"threads_started":1}
I20260429 14:03:42.899363 4486 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:42.893578 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37116 (local address 127.2.197.194:40607)
0429 14:03:42.895139 (+ 1561us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:42.895147 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:42.895158 (+ 11us) server_negotiation.cc:408] Connection header received
0429 14:03:42.895225 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:42.895227 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:42.895269 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:42.895358 (+ 89us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:03:42.895767 (+ 409us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.896912 (+ 1145us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:42.898464 (+ 1552us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:42.898952 (+ 488us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:42.898991 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:42.899231 (+ 240us) server_negotiation.cc:300] Negotiation successful
0429 14:03:42.899287 (+ 56us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1500,"thread_start_us":172,"threads_started":1}
I20260429 14:03:42.899541 4114 tablet_service.cc:1511] Processing CreateTablet for tablet 010d9785e99148d78bc921ce83beff6a (DEFAULT_TABLE table=test-table [id=38cf6dc750fa4de2a0774be0ce5d9e9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:03:42.899891 4114 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 010d9785e99148d78bc921ce83beff6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:03:42.900710 4487 tablet_bootstrap.cc:492] T 010d9785e99148d78bc921ce83beff6a P 52af864dbc164356a9ef9b9974c6be29: Bootstrap starting.
I20260429 14:03:42.901352 4487 tablet_bootstrap.cc:654] T 010d9785e99148d78bc921ce83beff6a P 52af864dbc164356a9ef9b9974c6be29: Neither blocks nor log segments found. Creating new log.
I20260429 14:03:42.901858 4487 log.cc:826] T 010d9785e99148d78bc921ce83beff6a P 52af864dbc164356a9ef9b9974c6be29: Log is configured to *not* fsync() on all Append() calls
I20260429 14:03:42.902654 4487 tablet_bootstrap.cc:492] T 010d9785e99148d78bc921ce83beff6a P 52af864dbc164356a9ef9b9974c6be29: No bootstrap required, opened a new log
I20260429 14:03:42.902737 4487 ts_tablet_manager.cc:1403] T 010d9785e99148d78bc921ce83beff6a P 52af864dbc164356a9ef9b9974c6be29: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260429 14:03:42.902707 4256 tablet_service.cc:1511] Processing CreateTablet for tablet 010d9785e99148d78bc921ce83beff6a (DEFAULT_TABLE table=test-table [id=38cf6dc750fa4de2a0774be0ce5d9e9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:03:42.902992 4256 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 010d9785e99148d78bc921ce83beff6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:03:42.903137 4489 tablet_bootstrap.cc:492] T 010d9785e99148d78bc921ce83beff6a P 6a2f154bcc534134bd963d22d54ba278: Bootstrap starting.
I20260429 14:03:42.903769 4489 tablet_bootstrap.cc:654] T 010d9785e99148d78bc921ce83beff6a P 6a2f154bcc534134bd963d22d54ba278: Neither blocks nor log segments found. Creating new log.
I20260429 14:03:42.904088 4487 raft_consensus.cc:359] T 010d9785e99148d78bc921ce83beff6a P 52af864dbc164356a9ef9b9974c6be29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } } peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } }
I20260429 14:03:42.904222 4487 raft_consensus.cc:385] T 010d9785e99148d78bc921ce83beff6a P 52af864dbc164356a9ef9b9974c6be29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:03:42.904258 4487 raft_consensus.cc:740] T 010d9785e99148d78bc921ce83beff6a P 52af864dbc164356a9ef9b9974c6be29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52af864dbc164356a9ef9b9974c6be29, State: Initialized, Role: FOLLOWER
I20260429 14:03:42.904347 4489 log.cc:826] T 010d9785e99148d78bc921ce83beff6a P 6a2f154bcc534134bd963d22d54ba278: Log is configured to *not* fsync() on all Append() calls
I20260429 14:03:42.904338 4487 consensus_queue.cc:260] T 010d9785e99148d78bc921ce83beff6a P 52af864dbc164356a9ef9b9974c6be29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } } peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } }
I20260429 14:03:42.904592 4487 ts_tablet_manager.cc:1434] T 010d9785e99148d78bc921ce83beff6a P 52af864dbc164356a9ef9b9974c6be29: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260429 14:03:42.904678 4464 heartbeater.cc:499] Master 127.2.197.254:38449 was elected leader, sending a full tablet report...
I20260429 14:03:42.905279 4489 tablet_bootstrap.cc:492] T 010d9785e99148d78bc921ce83beff6a P 6a2f154bcc534134bd963d22d54ba278: No bootstrap required, opened a new log
I20260429 14:03:42.905349 4489 ts_tablet_manager.cc:1403] T 010d9785e99148d78bc921ce83beff6a P 6a2f154bcc534134bd963d22d54ba278: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260429 14:03:42.906421 4492 tablet_bootstrap.cc:492] T 010d9785e99148d78bc921ce83beff6a P 7de3fbd2cbcf477fa2b3dfc27674e490: Bootstrap starting.
I20260429 14:03:42.906948 4492 tablet_bootstrap.cc:654] T 010d9785e99148d78bc921ce83beff6a P 7de3fbd2cbcf477fa2b3dfc27674e490: Neither blocks nor log segments found. Creating new log.
I20260429 14:03:42.906890 4489 raft_consensus.cc:359] T 010d9785e99148d78bc921ce83beff6a P 6a2f154bcc534134bd963d22d54ba278 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } } peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } }
I20260429 14:03:42.907003 4489 raft_consensus.cc:385] T 010d9785e99148d78bc921ce83beff6a P 6a2f154bcc534134bd963d22d54ba278 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:03:42.907049 4489 raft_consensus.cc:740] T 010d9785e99148d78bc921ce83beff6a P 6a2f154bcc534134bd963d22d54ba278 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a2f154bcc534134bd963d22d54ba278, State: Initialized, Role: FOLLOWER
I20260429 14:03:42.907172 4489 consensus_queue.cc:260] T 010d9785e99148d78bc921ce83beff6a P 6a2f154bcc534134bd963d22d54ba278 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } } peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } }
I20260429 14:03:42.907428 4489 ts_tablet_manager.cc:1434] T 010d9785e99148d78bc921ce83beff6a P 6a2f154bcc534134bd963d22d54ba278: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260429 14:03:42.907487 4180 heartbeater.cc:499] Master 127.2.197.254:38449 was elected leader, sending a full tablet report...
I20260429 14:03:42.907596 4492 log.cc:826] T 010d9785e99148d78bc921ce83beff6a P 7de3fbd2cbcf477fa2b3dfc27674e490: Log is configured to *not* fsync() on all Append() calls
I20260429 14:03:42.908419 4492 tablet_bootstrap.cc:492] T 010d9785e99148d78bc921ce83beff6a P 7de3fbd2cbcf477fa2b3dfc27674e490: No bootstrap required, opened a new log
I20260429 14:03:42.908483 4492 ts_tablet_manager.cc:1403] T 010d9785e99148d78bc921ce83beff6a P 7de3fbd2cbcf477fa2b3dfc27674e490: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260429 14:03:42.909575 4492 raft_consensus.cc:359] T 010d9785e99148d78bc921ce83beff6a P 7de3fbd2cbcf477fa2b3dfc27674e490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } } peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } }
I20260429 14:03:42.909715 4492 raft_consensus.cc:385] T 010d9785e99148d78bc921ce83beff6a P 7de3fbd2cbcf477fa2b3dfc27674e490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:03:42.909755 4492 raft_consensus.cc:740] T 010d9785e99148d78bc921ce83beff6a P 7de3fbd2cbcf477fa2b3dfc27674e490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7de3fbd2cbcf477fa2b3dfc27674e490, State: Initialized, Role: FOLLOWER
I20260429 14:03:42.909875 4492 consensus_queue.cc:260] T 010d9785e99148d78bc921ce83beff6a P 7de3fbd2cbcf477fa2b3dfc27674e490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } } peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } }
I20260429 14:03:42.910131 4492 ts_tablet_manager.cc:1434] T 010d9785e99148d78bc921ce83beff6a P 7de3fbd2cbcf477fa2b3dfc27674e490: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260429 14:03:42.910173 4322 heartbeater.cc:499] Master 127.2.197.254:38449 was elected leader, sending a full tablet report...
W20260429 14:03:42.912142 4323 tablet.cc:2404] T 010d9785e99148d78bc921ce83beff6a P 7de3fbd2cbcf477fa2b3dfc27674e490: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260429 14:03:42.973775 4181 tablet.cc:2404] T 010d9785e99148d78bc921ce83beff6a P 6a2f154bcc534134bd963d22d54ba278: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260429 14:03:43.105400 4465 tablet.cc:2404] T 010d9785e99148d78bc921ce83beff6a P 52af864dbc164356a9ef9b9974c6be29: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 14:03:43.187947 4493 raft_consensus.cc:493] T 010d9785e99148d78bc921ce83beff6a P 6a2f154bcc534134bd963d22d54ba278 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:03:43.188078 4493 raft_consensus.cc:515] T 010d9785e99148d78bc921ce83beff6a P 6a2f154bcc534134bd963d22d54ba278 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } } peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } }
I20260429 14:03:43.188586 4493 leader_election.cc:290] T 010d9785e99148d78bc921ce83beff6a P 6a2f154bcc534134bd963d22d54ba278 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52af864dbc164356a9ef9b9974c6be29 (127.2.197.195:33361), 7de3fbd2cbcf477fa2b3dfc27674e490 (127.2.197.194:40607)
I20260429 14:03:43.191203 4496 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:43.188887 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:33361 (local address 127.2.197.193:44303)
0429 14:03:43.189031 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:43.189043 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:43.189112 (+ 69us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:43.189338 (+ 226us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:43.189339 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:43.189341 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:03:43.189495 (+ 154us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:43.189497 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:43.190308 (+ 811us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:43.190309 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:43.190998 (+ 689us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:43.191003 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:43.191065 (+ 62us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:43.191069 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 14:03:43.191093 (+ 24us) client_negotiation.cc:241] Negotiation successful
0429 14:03:43.191130 (+ 37us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":99,"threads_started":1}
I20260429 14:03:43.191991 4497 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:43.189111 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:40607 (local address 127.2.197.193:56195)
0429 14:03:43.189229 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:43.189258 (+ 29us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:43.189301 (+ 43us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:43.189440 (+ 139us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:43.189441 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:43.189444 (+ 3us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:03:43.189763 (+ 319us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:43.189765 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:43.190706 (+ 941us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:43.190709 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:43.191633 (+ 924us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:43.191708 (+ 75us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:43.191812 (+ 104us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:43.191818 (+ 6us) client_negotiation.cc:770] Sending connection context
0429 14:03:43.191856 (+ 38us) client_negotiation.cc:241] Negotiation successful
0429 14:03:43.191918 (+ 62us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":51,"threads_started":1}
I20260429 14:03:43.192046 4484 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:43.188957 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:44303 (local address 127.2.197.195:33361)
0429 14:03:43.189056 (+ 99us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:43.189058 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:43.189110 (+ 52us) server_negotiation.cc:408] Connection header received
0429 14:03:43.189188 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:43.189189 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:43.189200 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:43.189283 (+ 83us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:03:43.189575 (+ 292us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:43.190189 (+ 614us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:43.191122 (+ 933us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:43.191862 (+ 740us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:43.191874 (+ 12us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:43.191900 (+ 26us) server_negotiation.cc:300] Negotiation successful
0429 14:03:43.191950 (+ 50us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260429 14:03:43.192348 4486 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:43.189218 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:56195 (local address 127.2.197.194:40607)
0429 14:03:43.189296 (+ 78us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:43.189298 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:43.189308 (+ 10us) server_negotiation.cc:408] Connection header received
0429 14:03:43.189356 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:43.189357 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:43.189366 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:43.189434 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:03:43.189877 (+ 443us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:43.190476 (+ 599us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:43.191813 (+ 1337us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:43.192224 (+ 411us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:43.192233 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:43.192256 (+ 23us) server_negotiation.cc:300] Negotiation successful
0429 14:03:43.192295 (+ 39us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260429 14:03:43.192319 4418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010d9785e99148d78bc921ce83beff6a" candidate_uuid: "6a2f154bcc534134bd963d22d54ba278" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52af864dbc164356a9ef9b9974c6be29" is_pre_election: true
I20260429 14:03:43.192507 4418 raft_consensus.cc:2468] T 010d9785e99148d78bc921ce83beff6a P 52af864dbc164356a9ef9b9974c6be29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a2f154bcc534134bd963d22d54ba278 in term 0.
I20260429 14:03:43.192695 4275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010d9785e99148d78bc921ce83beff6a" candidate_uuid: "6a2f154bcc534134bd963d22d54ba278" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" is_pre_election: true
I20260429 14:03:43.192731 4068 leader_election.cc:304] T 010d9785e99148d78bc921ce83beff6a P 6a2f154bcc534134bd963d22d54ba278 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52af864dbc164356a9ef9b9974c6be29, 6a2f154bcc534134bd963d22d54ba278; no voters:
I20260429 14:03:43.192833 4275 raft_consensus.cc:2468] T 010d9785e99148d78bc921ce83beff6a P 7de3fbd2cbcf477fa2b3dfc27674e490 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a2f154bcc534134bd963d22d54ba278 in term 0.
I20260429 14:03:43.192958 4493 raft_consensus.cc:2804] T 010d9785e99148d78bc921ce83beff6a P 6a2f154bcc534134bd963d22d54ba278 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 14:03:43.193033 4493 raft_consensus.cc:493] T 010d9785e99148d78bc921ce83beff6a P 6a2f154bcc534134bd963d22d54ba278 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 14:03:43.193084 4493 raft_consensus.cc:3060] T 010d9785e99148d78bc921ce83beff6a P 6a2f154bcc534134bd963d22d54ba278 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:03:43.193754 4493 raft_consensus.cc:515] T 010d9785e99148d78bc921ce83beff6a P 6a2f154bcc534134bd963d22d54ba278 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } } peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } }
I20260429 14:03:43.193914 4493 leader_election.cc:290] T 010d9785e99148d78bc921ce83beff6a P 6a2f154bcc534134bd963d22d54ba278 [CANDIDATE]: Term 1 election: Requested vote from peers 52af864dbc164356a9ef9b9974c6be29 (127.2.197.195:33361), 7de3fbd2cbcf477fa2b3dfc27674e490 (127.2.197.194:40607)
I20260429 14:03:43.194190 4275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010d9785e99148d78bc921ce83beff6a" candidate_uuid: "6a2f154bcc534134bd963d22d54ba278" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490"
I20260429 14:03:43.194192 4418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "010d9785e99148d78bc921ce83beff6a" candidate_uuid: "6a2f154bcc534134bd963d22d54ba278" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52af864dbc164356a9ef9b9974c6be29"
I20260429 14:03:43.194298 4275 raft_consensus.cc:3060] T 010d9785e99148d78bc921ce83beff6a P 7de3fbd2cbcf477fa2b3dfc27674e490 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:03:43.194299 4418 raft_consensus.cc:3060] T 010d9785e99148d78bc921ce83beff6a P 52af864dbc164356a9ef9b9974c6be29 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:03:43.195195 4418 raft_consensus.cc:2468] T 010d9785e99148d78bc921ce83beff6a P 52af864dbc164356a9ef9b9974c6be29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a2f154bcc534134bd963d22d54ba278 in term 1.
I20260429 14:03:43.195195 4275 raft_consensus.cc:2468] T 010d9785e99148d78bc921ce83beff6a P 7de3fbd2cbcf477fa2b3dfc27674e490 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a2f154bcc534134bd963d22d54ba278 in term 1.
I20260429 14:03:43.195457 4068 leader_election.cc:304] T 010d9785e99148d78bc921ce83beff6a P 6a2f154bcc534134bd963d22d54ba278 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a2f154bcc534134bd963d22d54ba278, 7de3fbd2cbcf477fa2b3dfc27674e490; no voters:
I20260429 14:03:43.195617 4493 raft_consensus.cc:2804] T 010d9785e99148d78bc921ce83beff6a P 6a2f154bcc534134bd963d22d54ba278 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:03:43.195779 4493 raft_consensus.cc:697] T 010d9785e99148d78bc921ce83beff6a P 6a2f154bcc534134bd963d22d54ba278 [term 1 LEADER]: Becoming Leader. State: Replica: 6a2f154bcc534134bd963d22d54ba278, State: Running, Role: LEADER
I20260429 14:03:43.195883 4493 consensus_queue.cc:237] T 010d9785e99148d78bc921ce83beff6a P 6a2f154bcc534134bd963d22d54ba278 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } } peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } }
I20260429 14:03:43.196700 3974 catalog_manager.cc:5671] T 010d9785e99148d78bc921ce83beff6a P 6a2f154bcc534134bd963d22d54ba278 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a2f154bcc534134bd963d22d54ba278 (127.2.197.193). New cstate: current_term: 1 leader_uuid: "6a2f154bcc534134bd963d22d54ba278" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } health_report { overall_health: UNKNOWN } } }
I20260429 14:03:43.260246 4485 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:43.257885 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42724 (local address 127.2.197.193:41417)
0429 14:03:43.257956 (+ 71us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:43.257958 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:43.257966 (+ 8us) server_negotiation.cc:408] Connection header received
0429 14:03:43.258003 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:43.258004 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:43.258017 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:43.258072 (+ 55us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:03:43.258401 (+ 329us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:43.258913 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:43.259515 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:43.259720 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:43.259752 (+ 32us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:03:43.259851 (+ 99us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:03:43.259921 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:43.260159 (+ 238us) server_negotiation.cc:300] Negotiation successful
0429 14:03:43.260185 (+ 26us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260429 14:03:43.262379 4418 raft_consensus.cc:1275] T 010d9785e99148d78bc921ce83beff6a P 52af864dbc164356a9ef9b9974c6be29 [term 1 FOLLOWER]: Refusing update from remote peer 6a2f154bcc534134bd963d22d54ba278: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:03:43.262379 4275 raft_consensus.cc:1275] T 010d9785e99148d78bc921ce83beff6a P 7de3fbd2cbcf477fa2b3dfc27674e490 [term 1 FOLLOWER]: Refusing update from remote peer 6a2f154bcc534134bd963d22d54ba278: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:03:43.262809 4493 consensus_queue.cc:1048] T 010d9785e99148d78bc921ce83beff6a P 6a2f154bcc534134bd963d22d54ba278 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:03:43.262965 4498 consensus_queue.cc:1048] T 010d9785e99148d78bc921ce83beff6a P 6a2f154bcc534134bd963d22d54ba278 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:03:43.265831 4503 mvcc.cc:204] Tried to move back new op lower bound from 7280522949679009792 to 7280522949410779136. Current Snapshot: MvccSnapshot[applied={T|T < 7280522949679009792}]
I20260429 14:03:43.269286 3974 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:03:43.269381 3974 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:03:43.270345 3974 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 5fc55c28f11d408ebaf8a4aac4312b83: Sending DeleteTablet for 3 replicas of tablet 010d9785e99148d78bc921ce83beff6a
I20260429 14:03:43.270715 4114 tablet_service.cc:1558] Processing DeleteTablet for tablet 010d9785e99148d78bc921ce83beff6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:03:43 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:42718
I20260429 14:03:43.270779 4256 tablet_service.cc:1558] Processing DeleteTablet for tablet 010d9785e99148d78bc921ce83beff6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:03:43 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:37116
I20260429 14:03:43.270805 4398 tablet_service.cc:1558] Processing DeleteTablet for tablet 010d9785e99148d78bc921ce83beff6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:03:43 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:48846
I20260429 14:03:43.270915 3974 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 14:03:43.271063 3974 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 14:03:43.271059 4513 tablet_replica.cc:333] T 010d9785e99148d78bc921ce83beff6a P 52af864dbc164356a9ef9b9974c6be29: stopping tablet replica
I20260429 14:03:43.271147 4513 raft_consensus.cc:2243] T 010d9785e99148d78bc921ce83beff6a P 52af864dbc164356a9ef9b9974c6be29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:03:43.271272 4513 raft_consensus.cc:2272] T 010d9785e99148d78bc921ce83beff6a P 52af864dbc164356a9ef9b9974c6be29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:03:43.271263 4511 tablet_replica.cc:333] T 010d9785e99148d78bc921ce83beff6a P 6a2f154bcc534134bd963d22d54ba278: stopping tablet replica
I20260429 14:03:43.271344 4511 raft_consensus.cc:2243] T 010d9785e99148d78bc921ce83beff6a P 6a2f154bcc534134bd963d22d54ba278 [term 1 LEADER]: Raft consensus shutting down.
I20260429 14:03:43.271447 4511 raft_consensus.cc:2272] T 010d9785e99148d78bc921ce83beff6a P 6a2f154bcc534134bd963d22d54ba278 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:03:43.271641 4512 tablet_replica.cc:333] T 010d9785e99148d78bc921ce83beff6a P 7de3fbd2cbcf477fa2b3dfc27674e490: stopping tablet replica
I20260429 14:03:43.272214 4511 ts_tablet_manager.cc:1916] T 010d9785e99148d78bc921ce83beff6a P 6a2f154bcc534134bd963d22d54ba278: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:03:43.272526 4114 tablet_service.cc:1511] Processing CreateTablet for tablet ec1905cab7d8450d9ee06d4e2463afa5 (DEFAULT_TABLE table=test-table [id=974ef469d98b43f9af5a7c93b1e50416]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:03:43.272670 4114 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec1905cab7d8450d9ee06d4e2463afa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:03:43.272733 4398 tablet_service.cc:1511] Processing CreateTablet for tablet ec1905cab7d8450d9ee06d4e2463afa5 (DEFAULT_TABLE table=test-table [id=974ef469d98b43f9af5a7c93b1e50416]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:03:43.272854 4398 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec1905cab7d8450d9ee06d4e2463afa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:03:43.273819 4489 tablet_bootstrap.cc:492] T ec1905cab7d8450d9ee06d4e2463afa5 P 6a2f154bcc534134bd963d22d54ba278: Bootstrap starting.
I20260429 14:03:43.274232 4489 tablet_bootstrap.cc:654] T ec1905cab7d8450d9ee06d4e2463afa5 P 6a2f154bcc534134bd963d22d54ba278: Neither blocks nor log segments found. Creating new log.
I20260429 14:03:43.274261 4511 ts_tablet_manager.cc:1929] T 010d9785e99148d78bc921ce83beff6a P 6a2f154bcc534134bd963d22d54ba278: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 14:03:43.274258 4487 tablet_bootstrap.cc:492] T ec1905cab7d8450d9ee06d4e2463afa5 P 52af864dbc164356a9ef9b9974c6be29: Bootstrap starting.
I20260429 14:03:43.274350 4511 log.cc:1199] T 010d9785e99148d78bc921ce83beff6a P 6a2f154bcc534134bd963d22d54ba278: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-0/wal/wals/010d9785e99148d78bc921ce83beff6a
I20260429 14:03:43.274641 4511 ts_tablet_manager.cc:1950] T 010d9785e99148d78bc921ce83beff6a P 6a2f154bcc534134bd963d22d54ba278: Deleting consensus metadata
I20260429 14:03:43.274722 4487 tablet_bootstrap.cc:654] T ec1905cab7d8450d9ee06d4e2463afa5 P 52af864dbc164356a9ef9b9974c6be29: Neither blocks nor log segments found. Creating new log.
I20260429 14:03:43.274890 4489 tablet_bootstrap.cc:492] T ec1905cab7d8450d9ee06d4e2463afa5 P 6a2f154bcc534134bd963d22d54ba278: No bootstrap required, opened a new log
I20260429 14:03:43.274923 4489 ts_tablet_manager.cc:1403] T ec1905cab7d8450d9ee06d4e2463afa5 P 6a2f154bcc534134bd963d22d54ba278: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260429 14:03:43.272644 4256 tablet_service.cc:1511] Processing CreateTablet for tablet ec1905cab7d8450d9ee06d4e2463afa5 (DEFAULT_TABLE table=test-table [id=974ef469d98b43f9af5a7c93b1e50416]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:03:43.275118 4256 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec1905cab7d8450d9ee06d4e2463afa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:03:43.275085 4489 raft_consensus.cc:359] T ec1905cab7d8450d9ee06d4e2463afa5 P 6a2f154bcc534134bd963d22d54ba278 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } } peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } }
I20260429 14:03:43.275131 3960 catalog_manager.cc:5002] TS 6a2f154bcc534134bd963d22d54ba278 (127.2.197.193:41417): tablet 010d9785e99148d78bc921ce83beff6a (table test-table [id=38cf6dc750fa4de2a0774be0ce5d9e9b]) successfully deleted
I20260429 14:03:43.275156 4489 raft_consensus.cc:385] T ec1905cab7d8450d9ee06d4e2463afa5 P 6a2f154bcc534134bd963d22d54ba278 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:03:43.275177 4489 raft_consensus.cc:740] T ec1905cab7d8450d9ee06d4e2463afa5 P 6a2f154bcc534134bd963d22d54ba278 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a2f154bcc534134bd963d22d54ba278, State: Initialized, Role: FOLLOWER
I20260429 14:03:43.275234 4489 consensus_queue.cc:260] T ec1905cab7d8450d9ee06d4e2463afa5 P 6a2f154bcc534134bd963d22d54ba278 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } } peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } }
I20260429 14:03:43.275347 4489 ts_tablet_manager.cc:1434] T ec1905cab7d8450d9ee06d4e2463afa5 P 6a2f154bcc534134bd963d22d54ba278: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260429 14:03:43.275545 4513 ts_tablet_manager.cc:1916] T 010d9785e99148d78bc921ce83beff6a P 52af864dbc164356a9ef9b9974c6be29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:03:43.275697 4487 tablet_bootstrap.cc:492] T ec1905cab7d8450d9ee06d4e2463afa5 P 52af864dbc164356a9ef9b9974c6be29: No bootstrap required, opened a new log
I20260429 14:03:43.275743 4487 ts_tablet_manager.cc:1403] T ec1905cab7d8450d9ee06d4e2463afa5 P 52af864dbc164356a9ef9b9974c6be29: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260429 14:03:43.276044 4487 raft_consensus.cc:359] T ec1905cab7d8450d9ee06d4e2463afa5 P 52af864dbc164356a9ef9b9974c6be29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } } peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } }
I20260429 14:03:43.276115 4487 raft_consensus.cc:385] T ec1905cab7d8450d9ee06d4e2463afa5 P 52af864dbc164356a9ef9b9974c6be29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:03:43.276134 4487 raft_consensus.cc:740] T ec1905cab7d8450d9ee06d4e2463afa5 P 52af864dbc164356a9ef9b9974c6be29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52af864dbc164356a9ef9b9974c6be29, State: Initialized, Role: FOLLOWER
I20260429 14:03:43.276173 4487 consensus_queue.cc:260] T ec1905cab7d8450d9ee06d4e2463afa5 P 52af864dbc164356a9ef9b9974c6be29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } } peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } }
I20260429 14:03:43.276255 4487 ts_tablet_manager.cc:1434] T ec1905cab7d8450d9ee06d4e2463afa5 P 52af864dbc164356a9ef9b9974c6be29: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260429 14:03:43.276542 4492 tablet_bootstrap.cc:492] T ec1905cab7d8450d9ee06d4e2463afa5 P 7de3fbd2cbcf477fa2b3dfc27674e490: Bootstrap starting.
I20260429 14:03:43.276589 4512 raft_consensus.cc:2243] T 010d9785e99148d78bc921ce83beff6a P 7de3fbd2cbcf477fa2b3dfc27674e490 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:03:43.276700 4512 raft_consensus.cc:2272] T 010d9785e99148d78bc921ce83beff6a P 7de3fbd2cbcf477fa2b3dfc27674e490 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:03:43.276959 4492 tablet_bootstrap.cc:654] T ec1905cab7d8450d9ee06d4e2463afa5 P 7de3fbd2cbcf477fa2b3dfc27674e490: Neither blocks nor log segments found. Creating new log.
I20260429 14:03:43.277174 4512 ts_tablet_manager.cc:1916] T 010d9785e99148d78bc921ce83beff6a P 7de3fbd2cbcf477fa2b3dfc27674e490: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:03:43.277185 4513 ts_tablet_manager.cc:1929] T 010d9785e99148d78bc921ce83beff6a P 52af864dbc164356a9ef9b9974c6be29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 14:03:43.277244 4513 log.cc:1199] T 010d9785e99148d78bc921ce83beff6a P 52af864dbc164356a9ef9b9974c6be29: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-2/wal/wals/010d9785e99148d78bc921ce83beff6a
I20260429 14:03:43.277482 4513 ts_tablet_manager.cc:1950] T 010d9785e99148d78bc921ce83beff6a P 52af864dbc164356a9ef9b9974c6be29: Deleting consensus metadata
I20260429 14:03:43.277870 3962 catalog_manager.cc:5002] TS 52af864dbc164356a9ef9b9974c6be29 (127.2.197.195:33361): tablet 010d9785e99148d78bc921ce83beff6a (table test-table [id=38cf6dc750fa4de2a0774be0ce5d9e9b]) successfully deleted
I20260429 14:03:43.278342 4492 tablet_bootstrap.cc:492] T ec1905cab7d8450d9ee06d4e2463afa5 P 7de3fbd2cbcf477fa2b3dfc27674e490: No bootstrap required, opened a new log
I20260429 14:03:43.278386 4492 ts_tablet_manager.cc:1403] T ec1905cab7d8450d9ee06d4e2463afa5 P 7de3fbd2cbcf477fa2b3dfc27674e490: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260429 14:03:43.278676 4492 raft_consensus.cc:359] T ec1905cab7d8450d9ee06d4e2463afa5 P 7de3fbd2cbcf477fa2b3dfc27674e490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } } peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } }
I20260429 14:03:43.278759 4492 raft_consensus.cc:385] T ec1905cab7d8450d9ee06d4e2463afa5 P 7de3fbd2cbcf477fa2b3dfc27674e490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:03:43.278784 4492 raft_consensus.cc:740] T ec1905cab7d8450d9ee06d4e2463afa5 P 7de3fbd2cbcf477fa2b3dfc27674e490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7de3fbd2cbcf477fa2b3dfc27674e490, State: Initialized, Role: FOLLOWER
I20260429 14:03:43.278841 4492 consensus_queue.cc:260] T ec1905cab7d8450d9ee06d4e2463afa5 P 7de3fbd2cbcf477fa2b3dfc27674e490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } } peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } }
I20260429 14:03:43.278952 4492 ts_tablet_manager.cc:1434] T ec1905cab7d8450d9ee06d4e2463afa5 P 7de3fbd2cbcf477fa2b3dfc27674e490: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260429 14:03:43.278981 4512 ts_tablet_manager.cc:1929] T 010d9785e99148d78bc921ce83beff6a P 7de3fbd2cbcf477fa2b3dfc27674e490: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 14:03:43.279032 4512 log.cc:1199] T 010d9785e99148d78bc921ce83beff6a P 7de3fbd2cbcf477fa2b3dfc27674e490: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-1/wal/wals/010d9785e99148d78bc921ce83beff6a
I20260429 14:03:43.279305 4512 ts_tablet_manager.cc:1950] T 010d9785e99148d78bc921ce83beff6a P 7de3fbd2cbcf477fa2b3dfc27674e490: Deleting consensus metadata
I20260429 14:03:43.279779 3962 catalog_manager.cc:5002] TS 7de3fbd2cbcf477fa2b3dfc27674e490 (127.2.197.194:40607): tablet 010d9785e99148d78bc921ce83beff6a (table test-table [id=38cf6dc750fa4de2a0774be0ce5d9e9b]) successfully deleted
I20260429 14:03:43.311096 4495 raft_consensus.cc:493] T ec1905cab7d8450d9ee06d4e2463afa5 P 7de3fbd2cbcf477fa2b3dfc27674e490 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:03:43.311216 4495 raft_consensus.cc:515] T ec1905cab7d8450d9ee06d4e2463afa5 P 7de3fbd2cbcf477fa2b3dfc27674e490 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } } peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } }
I20260429 14:03:43.311543 4495 leader_election.cc:290] T ec1905cab7d8450d9ee06d4e2463afa5 P 7de3fbd2cbcf477fa2b3dfc27674e490 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a2f154bcc534134bd963d22d54ba278 (127.2.197.193:41417), 52af864dbc164356a9ef9b9974c6be29 (127.2.197.195:33361)
I20260429 14:03:43.314103 4518 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:43.311716 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:41417 (local address 127.2.197.194:36213)
0429 14:03:43.311905 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:43.311919 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:43.312000 (+ 81us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:43.312165 (+ 165us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:43.312166 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:43.312168 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:03:43.312431 (+ 263us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:43.312433 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:43.313250 (+ 817us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:43.313251 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:43.313870 (+ 619us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:43.313871 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:43.313941 (+ 70us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:43.313946 (+ 5us) client_negotiation.cc:770] Sending connection context
0429 14:03:43.313977 (+ 31us) client_negotiation.cc:241] Negotiation successful
0429 14:03:43.314017 (+ 40us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":67,"threads_started":1}
I20260429 14:03:43.314586 4485 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:43.311892 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:36213 (local address 127.2.197.193:41417)
0429 14:03:43.311951 (+ 59us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:43.311953 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:43.311992 (+ 39us) server_negotiation.cc:408] Connection header received
0429 14:03:43.312072 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:43.312073 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:43.312084 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:43.312145 (+ 61us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:03:43.312520 (+ 375us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:43.313166 (+ 646us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:43.313960 (+ 794us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:43.314456 (+ 496us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:43.314464 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:43.314487 (+ 23us) server_negotiation.cc:300] Negotiation successful
0429 14:03:43.314520 (+ 33us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260429 14:03:43.314831 4134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec1905cab7d8450d9ee06d4e2463afa5" candidate_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a2f154bcc534134bd963d22d54ba278" is_pre_election: true
I20260429 14:03:43.314957 4134 raft_consensus.cc:2468] T ec1905cab7d8450d9ee06d4e2463afa5 P 6a2f154bcc534134bd963d22d54ba278 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7de3fbd2cbcf477fa2b3dfc27674e490 in term 0.
I20260429 14:03:43.315148 4208 leader_election.cc:304] T ec1905cab7d8450d9ee06d4e2463afa5 P 7de3fbd2cbcf477fa2b3dfc27674e490 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a2f154bcc534134bd963d22d54ba278, 7de3fbd2cbcf477fa2b3dfc27674e490; no voters:
I20260429 14:03:43.315279 4495 raft_consensus.cc:2804] T ec1905cab7d8450d9ee06d4e2463afa5 P 7de3fbd2cbcf477fa2b3dfc27674e490 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 14:03:43.315328 4495 raft_consensus.cc:493] T ec1905cab7d8450d9ee06d4e2463afa5 P 7de3fbd2cbcf477fa2b3dfc27674e490 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 14:03:43.315351 4495 raft_consensus.cc:3060] T ec1905cab7d8450d9ee06d4e2463afa5 P 7de3fbd2cbcf477fa2b3dfc27674e490 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:03:43.315418 4484 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:43.311921 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:54245 (local address 127.2.197.195:33361)
0429 14:03:43.311958 (+ 37us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:43.311961 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:43.312073 (+ 112us) server_negotiation.cc:408] Connection header received
0429 14:03:43.312178 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:43.312179 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:43.312187 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:43.312217 (+ 30us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:03:43.312944 (+ 727us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:43.313693 (+ 749us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:43.314757 (+ 1064us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:43.315208 (+ 451us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:43.315217 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:43.315314 (+ 97us) server_negotiation.cc:300] Negotiation successful
0429 14:03:43.315352 (+ 38us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260429 14:03:43.315472 4517 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:43.311716 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:33361 (local address 127.2.197.194:54245)
0429 14:03:43.312000 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:43.312010 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:43.312099 (+ 89us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:43.312282 (+ 183us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:43.312283 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:43.312284 (+ 1us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:03:43.312841 (+ 557us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:43.312844 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:43.313783 (+ 939us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:43.313784 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:43.314631 (+ 847us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:43.314655 (+ 24us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:43.315246 (+ 591us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:43.315250 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 14:03:43.315308 (+ 58us) client_negotiation.cc:241] Negotiation successful
0429 14:03:43.315351 (+ 43us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":10,"thread_start_us":77,"threads_started":1}
I20260429 14:03:43.315686 4418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec1905cab7d8450d9ee06d4e2463afa5" candidate_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52af864dbc164356a9ef9b9974c6be29" is_pre_election: true
I20260429 14:03:43.315771 4418 raft_consensus.cc:2468] T ec1905cab7d8450d9ee06d4e2463afa5 P 52af864dbc164356a9ef9b9974c6be29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7de3fbd2cbcf477fa2b3dfc27674e490 in term 0.
I20260429 14:03:43.316210 4495 raft_consensus.cc:515] T ec1905cab7d8450d9ee06d4e2463afa5 P 7de3fbd2cbcf477fa2b3dfc27674e490 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } } peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } }
I20260429 14:03:43.316386 4495 leader_election.cc:290] T ec1905cab7d8450d9ee06d4e2463afa5 P 7de3fbd2cbcf477fa2b3dfc27674e490 [CANDIDATE]: Term 1 election: Requested vote from peers 6a2f154bcc534134bd963d22d54ba278 (127.2.197.193:41417), 52af864dbc164356a9ef9b9974c6be29 (127.2.197.195:33361)
I20260429 14:03:43.316571 4134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec1905cab7d8450d9ee06d4e2463afa5" candidate_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a2f154bcc534134bd963d22d54ba278"
I20260429 14:03:43.316648 4134 raft_consensus.cc:3060] T ec1905cab7d8450d9ee06d4e2463afa5 P 6a2f154bcc534134bd963d22d54ba278 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:03:43.316609 4418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec1905cab7d8450d9ee06d4e2463afa5" candidate_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52af864dbc164356a9ef9b9974c6be29"
I20260429 14:03:43.316707 4418 raft_consensus.cc:3060] T ec1905cab7d8450d9ee06d4e2463afa5 P 52af864dbc164356a9ef9b9974c6be29 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:03:43.317402 4418 raft_consensus.cc:2468] T ec1905cab7d8450d9ee06d4e2463afa5 P 52af864dbc164356a9ef9b9974c6be29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7de3fbd2cbcf477fa2b3dfc27674e490 in term 1.
I20260429 14:03:43.317435 4134 raft_consensus.cc:2468] T ec1905cab7d8450d9ee06d4e2463afa5 P 6a2f154bcc534134bd963d22d54ba278 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7de3fbd2cbcf477fa2b3dfc27674e490 in term 1.
I20260429 14:03:43.317600 4208 leader_election.cc:304] T ec1905cab7d8450d9ee06d4e2463afa5 P 7de3fbd2cbcf477fa2b3dfc27674e490 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a2f154bcc534134bd963d22d54ba278, 7de3fbd2cbcf477fa2b3dfc27674e490; no voters:
I20260429 14:03:43.317677 4495 raft_consensus.cc:2804] T ec1905cab7d8450d9ee06d4e2463afa5 P 7de3fbd2cbcf477fa2b3dfc27674e490 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:03:43.317883 4495 raft_consensus.cc:697] T ec1905cab7d8450d9ee06d4e2463afa5 P 7de3fbd2cbcf477fa2b3dfc27674e490 [term 1 LEADER]: Becoming Leader. State: Replica: 7de3fbd2cbcf477fa2b3dfc27674e490, State: Running, Role: LEADER
I20260429 14:03:43.317960 4495 consensus_queue.cc:237] T ec1905cab7d8450d9ee06d4e2463afa5 P 7de3fbd2cbcf477fa2b3dfc27674e490 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } } peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } }
I20260429 14:03:43.318645 3974 catalog_manager.cc:5671] T ec1905cab7d8450d9ee06d4e2463afa5 P 7de3fbd2cbcf477fa2b3dfc27674e490 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7de3fbd2cbcf477fa2b3dfc27674e490 (127.2.197.194). New cstate: current_term: 1 leader_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } health_report { overall_health: UNKNOWN } } }
Apr 29 14:03:43 dist-test-slave-8n77 krb5kdc[3927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471422, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:03:43.342959 4024 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:43.335328 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49806 (local address 127.2.197.254:38449)
0429 14:03:43.335379 (+ 51us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:43.335381 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:43.335512 (+ 131us) server_negotiation.cc:408] Connection header received
0429 14:03:43.335551 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:43.335552 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:43.335568 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:43.335630 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:43.336493 (+ 863us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:43.337014 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:43.337800 (+ 786us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:43.337916 (+ 116us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:43.340195 (+ 2279us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:43.340200 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:43.340200 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:43.340209 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:43.341601 (+ 1392us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:43.342007 (+ 406us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:43.342008 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:43.342009 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:43.342033 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:43.342197 (+ 164us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:43.342197 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:43.342198 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:43.342357 (+ 159us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:43.342431 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:43.342808 (+ 377us) server_negotiation.cc:300] Negotiation successful
0429 14:03:43.342890 (+ 82us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260429 14:03:43.343112 4526 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:43.335217 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:38449 (local address 127.0.0.1:49806)
0429 14:03:43.335390 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:43.335403 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:43.335493 (+ 90us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:43.335654 (+ 161us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:43.335655 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:43.336148 (+ 493us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:03:43.336385 (+ 237us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:43.336388 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:43.337100 (+ 712us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:43.337101 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:43.337701 (+ 600us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:43.337704 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:43.337783 (+ 79us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:43.338334 (+ 551us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:03:43.338339 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:03:43.340096 (+ 1757us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:03:43.341712 (+ 1616us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:43.341718 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:43.341721 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:43.341920 (+ 199us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:43.342100 (+ 180us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:43.342101 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:43.342101 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:43.342138 (+ 37us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:43.342630 (+ 492us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:03:43.342635 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:03:43.342688 (+ 53us) client_negotiation.cc:770] Sending connection context
0429 14:03:43.342776 (+ 88us) client_negotiation.cc:241] Negotiation successful
0429 14:03:43.342993 (+ 217us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":82,"thread_start_us":91,"threads_started":1}
I20260429 14:03:43.345369 3974 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:49806:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260429 14:03:43.345611 3974 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 14:03:43.347141 4256 tablet_service.cc:1511] Processing CreateTablet for tablet 798364a5718948c39cc1334afd310629 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4ac69de7b3ea410a876326044814e818]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260429 14:03:43.347121 4114 tablet_service.cc:1511] Processing CreateTablet for tablet 798364a5718948c39cc1334afd310629 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4ac69de7b3ea410a876326044814e818]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260429 14:03:43.347276 4256 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 798364a5718948c39cc1334afd310629. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:03:43.347276 4114 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 798364a5718948c39cc1334afd310629. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:03:43.348806 4489 tablet_bootstrap.cc:492] T 798364a5718948c39cc1334afd310629 P 6a2f154bcc534134bd963d22d54ba278: Bootstrap starting.
I20260429 14:03:43.348876 4398 tablet_service.cc:1511] Processing CreateTablet for tablet 798364a5718948c39cc1334afd310629 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4ac69de7b3ea410a876326044814e818]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260429 14:03:43.349012 4398 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 798364a5718948c39cc1334afd310629. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:03:43.349471 4489 tablet_bootstrap.cc:654] T 798364a5718948c39cc1334afd310629 P 6a2f154bcc534134bd963d22d54ba278: Neither blocks nor log segments found. Creating new log.
I20260429 14:03:43.350157 4492 tablet_bootstrap.cc:492] T 798364a5718948c39cc1334afd310629 P 7de3fbd2cbcf477fa2b3dfc27674e490: Bootstrap starting.
I20260429 14:03:43.350224 4489 tablet_bootstrap.cc:492] T 798364a5718948c39cc1334afd310629 P 6a2f154bcc534134bd963d22d54ba278: No bootstrap required, opened a new log
I20260429 14:03:43.350267 4489 ts_tablet_manager.cc:1403] T 798364a5718948c39cc1334afd310629 P 6a2f154bcc534134bd963d22d54ba278: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260429 14:03:43.350441 4489 raft_consensus.cc:359] T 798364a5718948c39cc1334afd310629 P 6a2f154bcc534134bd963d22d54ba278 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } } peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } }
I20260429 14:03:43.350502 4489 raft_consensus.cc:385] T 798364a5718948c39cc1334afd310629 P 6a2f154bcc534134bd963d22d54ba278 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:03:43.350522 4489 raft_consensus.cc:740] T 798364a5718948c39cc1334afd310629 P 6a2f154bcc534134bd963d22d54ba278 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a2f154bcc534134bd963d22d54ba278, State: Initialized, Role: FOLLOWER
I20260429 14:03:43.350571 4489 consensus_queue.cc:260] T 798364a5718948c39cc1334afd310629 P 6a2f154bcc534134bd963d22d54ba278 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } } peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } }
I20260429 14:03:43.350648 4492 tablet_bootstrap.cc:654] T 798364a5718948c39cc1334afd310629 P 7de3fbd2cbcf477fa2b3dfc27674e490: Neither blocks nor log segments found. Creating new log.
I20260429 14:03:43.350669 4489 ts_tablet_manager.cc:1434] T 798364a5718948c39cc1334afd310629 P 6a2f154bcc534134bd963d22d54ba278: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:43.350745 4506 tablet_replica.cc:442] T 798364a5718948c39cc1334afd310629 P 6a2f154bcc534134bd963d22d54ba278: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } } peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } } }
I20260429 14:03:43.350841 4506 tablet_replica.cc:445] T 798364a5718948c39cc1334afd310629 P 6a2f154bcc534134bd963d22d54ba278: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 14:03:43.351028 4487 tablet_bootstrap.cc:492] T 798364a5718948c39cc1334afd310629 P 52af864dbc164356a9ef9b9974c6be29: Bootstrap starting.
I20260429 14:03:43.351482 4487 tablet_bootstrap.cc:654] T 798364a5718948c39cc1334afd310629 P 52af864dbc164356a9ef9b9974c6be29: Neither blocks nor log segments found. Creating new log.
I20260429 14:03:43.352183 4487 tablet_bootstrap.cc:492] T 798364a5718948c39cc1334afd310629 P 52af864dbc164356a9ef9b9974c6be29: No bootstrap required, opened a new log
I20260429 14:03:43.352244 4487 ts_tablet_manager.cc:1403] T 798364a5718948c39cc1334afd310629 P 52af864dbc164356a9ef9b9974c6be29: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260429 14:03:43.352398 4487 raft_consensus.cc:359] T 798364a5718948c39cc1334afd310629 P 52af864dbc164356a9ef9b9974c6be29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } } peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } }
I20260429 14:03:43.352474 4487 raft_consensus.cc:385] T 798364a5718948c39cc1334afd310629 P 52af864dbc164356a9ef9b9974c6be29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:03:43.352495 4487 raft_consensus.cc:740] T 798364a5718948c39cc1334afd310629 P 52af864dbc164356a9ef9b9974c6be29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52af864dbc164356a9ef9b9974c6be29, State: Initialized, Role: FOLLOWER
I20260429 14:03:43.352550 4487 consensus_queue.cc:260] T 798364a5718948c39cc1334afd310629 P 52af864dbc164356a9ef9b9974c6be29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } } peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } }
I20260429 14:03:43.352640 4487 ts_tablet_manager.cc:1434] T 798364a5718948c39cc1334afd310629 P 52af864dbc164356a9ef9b9974c6be29: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260429 14:03:43.352751 4492 tablet_bootstrap.cc:492] T 798364a5718948c39cc1334afd310629 P 7de3fbd2cbcf477fa2b3dfc27674e490: No bootstrap required, opened a new log
I20260429 14:03:43.352808 4492 ts_tablet_manager.cc:1403] T 798364a5718948c39cc1334afd310629 P 7de3fbd2cbcf477fa2b3dfc27674e490: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260429 14:03:43.352767 4490 tablet_replica.cc:442] T 798364a5718948c39cc1334afd310629 P 52af864dbc164356a9ef9b9974c6be29: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } } peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } } }
I20260429 14:03:43.352859 4490 tablet_replica.cc:445] T 798364a5718948c39cc1334afd310629 P 52af864dbc164356a9ef9b9974c6be29: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 14:03:43.352988 4492 raft_consensus.cc:359] T 798364a5718948c39cc1334afd310629 P 7de3fbd2cbcf477fa2b3dfc27674e490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } } peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } }
I20260429 14:03:43.353075 4492 raft_consensus.cc:385] T 798364a5718948c39cc1334afd310629 P 7de3fbd2cbcf477fa2b3dfc27674e490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:03:43.353103 4492 raft_consensus.cc:740] T 798364a5718948c39cc1334afd310629 P 7de3fbd2cbcf477fa2b3dfc27674e490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7de3fbd2cbcf477fa2b3dfc27674e490, State: Initialized, Role: FOLLOWER
I20260429 14:03:43.353165 4492 consensus_queue.cc:260] T 798364a5718948c39cc1334afd310629 P 7de3fbd2cbcf477fa2b3dfc27674e490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } } peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } }
I20260429 14:03:43.353250 4492 ts_tablet_manager.cc:1434] T 798364a5718948c39cc1334afd310629 P 7de3fbd2cbcf477fa2b3dfc27674e490: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260429 14:03:43.353307 4495 tablet_replica.cc:442] T 798364a5718948c39cc1334afd310629 P 7de3fbd2cbcf477fa2b3dfc27674e490: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } } peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } } }
I20260429 14:03:43.353396 4495 tablet_replica.cc:445] T 798364a5718948c39cc1334afd310629 P 7de3fbd2cbcf477fa2b3dfc27674e490: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 14:03:43.426497 4495 raft_consensus.cc:493] T 798364a5718948c39cc1334afd310629 P 7de3fbd2cbcf477fa2b3dfc27674e490 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:03:43.426630 4495 raft_consensus.cc:515] T 798364a5718948c39cc1334afd310629 P 7de3fbd2cbcf477fa2b3dfc27674e490 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } } peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } }
I20260429 14:03:43.426841 4495 leader_election.cc:290] T 798364a5718948c39cc1334afd310629 P 7de3fbd2cbcf477fa2b3dfc27674e490 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a2f154bcc534134bd963d22d54ba278 (127.2.197.193:41417), 52af864dbc164356a9ef9b9974c6be29 (127.2.197.195:33361)
I20260429 14:03:43.426999 4134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "798364a5718948c39cc1334afd310629" candidate_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a2f154bcc534134bd963d22d54ba278" is_pre_election: true
I20260429 14:03:43.427089 4134 raft_consensus.cc:2468] T 798364a5718948c39cc1334afd310629 P 6a2f154bcc534134bd963d22d54ba278 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7de3fbd2cbcf477fa2b3dfc27674e490 in term 0.
I20260429 14:03:43.427057 4418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "798364a5718948c39cc1334afd310629" candidate_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52af864dbc164356a9ef9b9974c6be29" is_pre_election: true
I20260429 14:03:43.427131 4418 raft_consensus.cc:2468] T 798364a5718948c39cc1334afd310629 P 52af864dbc164356a9ef9b9974c6be29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7de3fbd2cbcf477fa2b3dfc27674e490 in term 0.
I20260429 14:03:43.427237 4208 leader_election.cc:304] T 798364a5718948c39cc1334afd310629 P 7de3fbd2cbcf477fa2b3dfc27674e490 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a2f154bcc534134bd963d22d54ba278, 7de3fbd2cbcf477fa2b3dfc27674e490; no voters:
I20260429 14:03:43.427361 4495 raft_consensus.cc:2804] T 798364a5718948c39cc1334afd310629 P 7de3fbd2cbcf477fa2b3dfc27674e490 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 14:03:43.427423 4495 raft_consensus.cc:493] T 798364a5718948c39cc1334afd310629 P 7de3fbd2cbcf477fa2b3dfc27674e490 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 14:03:43.427456 4495 raft_consensus.cc:3060] T 798364a5718948c39cc1334afd310629 P 7de3fbd2cbcf477fa2b3dfc27674e490 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:03:43.428104 4495 raft_consensus.cc:515] T 798364a5718948c39cc1334afd310629 P 7de3fbd2cbcf477fa2b3dfc27674e490 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } } peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } }
I20260429 14:03:43.428257 4495 leader_election.cc:290] T 798364a5718948c39cc1334afd310629 P 7de3fbd2cbcf477fa2b3dfc27674e490 [CANDIDATE]: Term 1 election: Requested vote from peers 6a2f154bcc534134bd963d22d54ba278 (127.2.197.193:41417), 52af864dbc164356a9ef9b9974c6be29 (127.2.197.195:33361)
I20260429 14:03:43.428447 4418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "798364a5718948c39cc1334afd310629" candidate_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52af864dbc164356a9ef9b9974c6be29"
I20260429 14:03:43.428524 4418 raft_consensus.cc:3060] T 798364a5718948c39cc1334afd310629 P 52af864dbc164356a9ef9b9974c6be29 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:03:43.429198 4418 raft_consensus.cc:2468] T 798364a5718948c39cc1334afd310629 P 52af864dbc164356a9ef9b9974c6be29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7de3fbd2cbcf477fa2b3dfc27674e490 in term 1.
I20260429 14:03:43.429333 4134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "798364a5718948c39cc1334afd310629" candidate_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a2f154bcc534134bd963d22d54ba278"
I20260429 14:03:43.429412 4210 leader_election.cc:304] T 798364a5718948c39cc1334afd310629 P 7de3fbd2cbcf477fa2b3dfc27674e490 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52af864dbc164356a9ef9b9974c6be29, 7de3fbd2cbcf477fa2b3dfc27674e490; no voters:
I20260429 14:03:43.429440 4134 raft_consensus.cc:3060] T 798364a5718948c39cc1334afd310629 P 6a2f154bcc534134bd963d22d54ba278 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:03:43.429493 4495 raft_consensus.cc:2804] T 798364a5718948c39cc1334afd310629 P 7de3fbd2cbcf477fa2b3dfc27674e490 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:03:43.429538 4495 raft_consensus.cc:697] T 798364a5718948c39cc1334afd310629 P 7de3fbd2cbcf477fa2b3dfc27674e490 [term 1 LEADER]: Becoming Leader. State: Replica: 7de3fbd2cbcf477fa2b3dfc27674e490, State: Running, Role: LEADER
I20260429 14:03:43.429594 4495 consensus_queue.cc:237] T 798364a5718948c39cc1334afd310629 P 7de3fbd2cbcf477fa2b3dfc27674e490 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } } peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } }
I20260429 14:03:43.429885 4495 tablet_replica.cc:442] T 798364a5718948c39cc1334afd310629 P 7de3fbd2cbcf477fa2b3dfc27674e490: TxnStatusTablet state changed. Reason: New leader 7de3fbd2cbcf477fa2b3dfc27674e490. Latest consensus state: current_term: 1 leader_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } } peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } } }
I20260429 14:03:43.429942 4495 tablet_replica.cc:445] T 798364a5718948c39cc1334afd310629 P 7de3fbd2cbcf477fa2b3dfc27674e490: This TxnStatusTablet replica's current role is: LEADER
I20260429 14:03:43.430253 4134 raft_consensus.cc:2468] T 798364a5718948c39cc1334afd310629 P 6a2f154bcc534134bd963d22d54ba278 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7de3fbd2cbcf477fa2b3dfc27674e490 in term 1.
I20260429 14:03:43.430440 3974 catalog_manager.cc:5671] T 798364a5718948c39cc1334afd310629 P 7de3fbd2cbcf477fa2b3dfc27674e490 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7de3fbd2cbcf477fa2b3dfc27674e490 (127.2.197.194). New cstate: current_term: 1 leader_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } health_report { overall_health: UNKNOWN } } }
I20260429 14:03:43.430560 4531 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260429 14:03:43.430648 4531 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260429 14:03:43.447916 4526 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:43.444978 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:40607 (local address 127.0.0.1:37120)
0429 14:03:43.445173 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:43.445193 (+ 20us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:43.445324 (+ 131us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:43.445557 (+ 233us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:43.445558 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:43.445560 (+ 2us) client_negotiation.cc:190] Negotiated authn=TOKEN
0429 14:03:43.445937 (+ 377us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:43.445942 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:43.446729 (+ 787us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:43.446731 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:43.447153 (+ 422us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:43.447154 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:43.447260 (+ 106us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:43.447279 (+ 19us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0429 14:03:43.447708 (+ 429us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0429 14:03:43.447716 (+ 8us) client_negotiation.cc:770] Sending connection context
0429 14:03:43.447794 (+ 78us) client_negotiation.cc:241] Negotiation successful
0429 14:03:43.447850 (+ 56us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260429 14:03:43.447908 4486 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:43.445256 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37120 (local address 127.2.197.194:40607)
0429 14:03:43.445338 (+ 82us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:43.445341 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:43.445385 (+ 44us) server_negotiation.cc:408] Connection header received
0429 14:03:43.445457 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:43.445459 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:43.445475 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:43.445544 (+ 69us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:03:43.446068 (+ 524us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:43.446632 (+ 564us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:43.447264 (+ 632us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:43.447449 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:43.447523 (+ 74us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:03:43.447609 (+ 86us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:03:43.447696 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:43.447800 (+ 104us) server_negotiation.cc:300] Negotiation successful
0429 14:03:43.447827 (+ 27us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260429 14:03:43.449091 4418 raft_consensus.cc:1275] T 798364a5718948c39cc1334afd310629 P 52af864dbc164356a9ef9b9974c6be29 [term 1 FOLLOWER]: Refusing update from remote peer 7de3fbd2cbcf477fa2b3dfc27674e490: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:03:43.449229 4134 raft_consensus.cc:1275] T 798364a5718948c39cc1334afd310629 P 6a2f154bcc534134bd963d22d54ba278 [term 1 FOLLOWER]: Refusing update from remote peer 7de3fbd2cbcf477fa2b3dfc27674e490: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:03:43.449375 4519 consensus_queue.cc:1048] T 798364a5718948c39cc1334afd310629 P 7de3fbd2cbcf477fa2b3dfc27674e490 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:03:43.449501 4495 consensus_queue.cc:1048] T 798364a5718948c39cc1334afd310629 P 7de3fbd2cbcf477fa2b3dfc27674e490 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:03:43.450104 4490 tablet_replica.cc:442] T 798364a5718948c39cc1334afd310629 P 52af864dbc164356a9ef9b9974c6be29: TxnStatusTablet state changed. Reason: New leader 7de3fbd2cbcf477fa2b3dfc27674e490. Latest consensus state: current_term: 1 leader_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } } peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } } }
I20260429 14:03:43.450191 4490 tablet_replica.cc:445] T 798364a5718948c39cc1334afd310629 P 52af864dbc164356a9ef9b9974c6be29: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 14:03:43.450737 4506 tablet_replica.cc:442] T 798364a5718948c39cc1334afd310629 P 6a2f154bcc534134bd963d22d54ba278: TxnStatusTablet state changed. Reason: New leader 7de3fbd2cbcf477fa2b3dfc27674e490. Latest consensus state: current_term: 1 leader_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } } peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } } }
I20260429 14:03:43.450831 4506 tablet_replica.cc:445] T 798364a5718948c39cc1334afd310629 P 6a2f154bcc534134bd963d22d54ba278: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 14:03:43.451607 4536 tablet_replica.cc:442] T 798364a5718948c39cc1334afd310629 P 7de3fbd2cbcf477fa2b3dfc27674e490: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } } peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } } }
I20260429 14:03:43.451759 4536 tablet_replica.cc:445] T 798364a5718948c39cc1334afd310629 P 7de3fbd2cbcf477fa2b3dfc27674e490: This TxnStatusTablet replica's current role is: LEADER
I20260429 14:03:43.451768 4495 tablet_replica.cc:442] T 798364a5718948c39cc1334afd310629 P 7de3fbd2cbcf477fa2b3dfc27674e490: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } } peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } } }
I20260429 14:03:43.452018 4495 tablet_replica.cc:445] T 798364a5718948c39cc1334afd310629 P 7de3fbd2cbcf477fa2b3dfc27674e490: This TxnStatusTablet replica's current role is: LEADER
I20260429 14:03:43.451928 4502 mvcc.cc:204] Tried to move back new op lower bound from 7280522950445350912 to 7280522950367965184. Current Snapshot: MvccSnapshot[applied={T|T < 7280522950445350912 or (T in {7280522950445350912})}]
I20260429 14:03:43.452775 4501 mvcc.cc:204] Tried to move back new op lower bound from 7280522950445350912 to 7280522950367965184. Current Snapshot: MvccSnapshot[applied={T|T < 7280522950445350912 or (T in {7280522950445350912})}]
I20260429 14:03:43.453006 4506 tablet_replica.cc:442] T 798364a5718948c39cc1334afd310629 P 6a2f154bcc534134bd963d22d54ba278: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } } peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } } }
I20260429 14:03:43.453085 4506 tablet_replica.cc:445] T 798364a5718948c39cc1334afd310629 P 6a2f154bcc534134bd963d22d54ba278: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 14:03:43.453859 4490 tablet_replica.cc:442] T 798364a5718948c39cc1334afd310629 P 52af864dbc164356a9ef9b9974c6be29: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 } } peers { permanent_uuid: "7de3fbd2cbcf477fa2b3dfc27674e490" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40607 } } peers { permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 } } }
I20260429 14:03:43.453946 4490 tablet_replica.cc:445] T 798364a5718948c39cc1334afd310629 P 52af864dbc164356a9ef9b9974c6be29: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 14:03:43.456154 4486 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:43.453511 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37126 (local address 127.2.197.194:40607)
0429 14:03:43.453549 (+ 38us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:43.453551 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:43.453558 (+ 7us) server_negotiation.cc:408] Connection header received
0429 14:03:43.453591 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:43.453592 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:43.453603 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:43.453654 (+ 51us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:03:43.454085 (+ 431us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:43.454717 (+ 632us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:43.455619 (+ 902us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:43.455740 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:43.455758 (+ 18us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:03:43.455779 (+ 21us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:03:43.455849 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:43.455937 (+ 88us) server_negotiation.cc:300] Negotiation successful
0429 14:03:43.455958 (+ 21us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260429 14:03:43.461342 4486 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:43.458912 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:42825 (local address 127.2.197.194:40607)
0429 14:03:43.458953 (+ 41us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:43.458956 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:43.459080 (+ 124us) server_negotiation.cc:408] Connection header received
0429 14:03:43.459141 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:43.459141 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:43.459149 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:43.459195 (+ 46us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:03:43.459581 (+ 386us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:43.460160 (+ 579us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:43.460972 (+ 812us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:43.461078 (+ 106us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:43.461093 (+ 15us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:03:43.461112 (+ 19us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:03:43.461156 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:43.461255 (+ 99us) server_negotiation.cc:300] Negotiation successful
0429 14:03:43.461274 (+ 19us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260429 14:03:43.461431 4539 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:43.458860 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:40607 (local address 127.2.197.194:42825)
0429 14:03:43.458999 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:43.459011 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:43.459081 (+ 70us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:43.459217 (+ 136us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:43.459218 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:43.459221 (+ 3us) client_negotiation.cc:190] Negotiated authn=TOKEN
0429 14:03:43.459512 (+ 291us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:43.459514 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:43.460237 (+ 723us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:43.460238 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:43.460889 (+ 651us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:43.460892 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:43.460970 (+ 78us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:43.460986 (+ 16us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0429 14:03:43.461185 (+ 199us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0429 14:03:43.461187 (+ 2us) client_negotiation.cc:770] Sending connection context
0429 14:03:43.461241 (+ 54us) client_negotiation.cc:241] Negotiation successful
0429 14:03:43.461281 (+ 40us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
I20260429 14:03:43.464282 4418 raft_consensus.cc:1275] T ec1905cab7d8450d9ee06d4e2463afa5 P 52af864dbc164356a9ef9b9974c6be29 [term 1 FOLLOWER]: Refusing update from remote peer 7de3fbd2cbcf477fa2b3dfc27674e490: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:03:43.464318 4134 raft_consensus.cc:1275] T ec1905cab7d8450d9ee06d4e2463afa5 P 6a2f154bcc534134bd963d22d54ba278 [term 1 FOLLOWER]: Refusing update from remote peer 7de3fbd2cbcf477fa2b3dfc27674e490: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:03:43.464524 4536 consensus_queue.cc:1048] T ec1905cab7d8450d9ee06d4e2463afa5 P 7de3fbd2cbcf477fa2b3dfc27674e490 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a2f154bcc534134bd963d22d54ba278" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:03:43.464620 4519 consensus_queue.cc:1048] T ec1905cab7d8450d9ee06d4e2463afa5 P 7de3fbd2cbcf477fa2b3dfc27674e490 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52af864dbc164356a9ef9b9974c6be29" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 33361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:03:43.482163 3975 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:03:43.482332 3975 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:03:43.483641 3975 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 5fc55c28f11d408ebaf8a4aac4312b83: Sending DeleteTablet for 3 replicas of tablet ec1905cab7d8450d9ee06d4e2463afa5
I20260429 14:03:43.484190 4256 tablet_service.cc:1558] Processing DeleteTablet for tablet ec1905cab7d8450d9ee06d4e2463afa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:03:43 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:37116
I20260429 14:03:43.484133 4398 tablet_service.cc:1558] Processing DeleteTablet for tablet ec1905cab7d8450d9ee06d4e2463afa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:03:43 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:48846
I20260429 14:03:43.484323 4512 tablet_replica.cc:333] T ec1905cab7d8450d9ee06d4e2463afa5 P 7de3fbd2cbcf477fa2b3dfc27674e490: stopping tablet replica
I20260429 14:03:43.484365 4114 tablet_service.cc:1558] Processing DeleteTablet for tablet ec1905cab7d8450d9ee06d4e2463afa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:03:43 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:42718
I20260429 14:03:43.484392 4512 raft_consensus.cc:2243] T ec1905cab7d8450d9ee06d4e2463afa5 P 7de3fbd2cbcf477fa2b3dfc27674e490 [term 1 LEADER]: Raft consensus shutting down.
I20260429 14:03:43.484458 4513 tablet_replica.cc:333] T ec1905cab7d8450d9ee06d4e2463afa5 P 52af864dbc164356a9ef9b9974c6be29: stopping tablet replica
I20260429 14:03:43.484584 4513 raft_consensus.cc:2243] T ec1905cab7d8450d9ee06d4e2463afa5 P 52af864dbc164356a9ef9b9974c6be29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:03:43.484625 4512 raft_consensus.cc:2272] T ec1905cab7d8450d9ee06d4e2463afa5 P 7de3fbd2cbcf477fa2b3dfc27674e490 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:03:43.484669 4513 raft_consensus.cc:2272] T ec1905cab7d8450d9ee06d4e2463afa5 P 52af864dbc164356a9ef9b9974c6be29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:03:43.484733 4511 tablet_replica.cc:333] T ec1905cab7d8450d9ee06d4e2463afa5 P 6a2f154bcc534134bd963d22d54ba278: stopping tablet replica
I20260429 14:03:43.484784 4511 raft_consensus.cc:2243] T ec1905cab7d8450d9ee06d4e2463afa5 P 6a2f154bcc534134bd963d22d54ba278 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:03:43.484826 4511 raft_consensus.cc:2272] T ec1905cab7d8450d9ee06d4e2463afa5 P 6a2f154bcc534134bd963d22d54ba278 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:03:43.485170 4512 ts_tablet_manager.cc:1916] T ec1905cab7d8450d9ee06d4e2463afa5 P 7de3fbd2cbcf477fa2b3dfc27674e490: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:03:43.485175 4511 ts_tablet_manager.cc:1916] T ec1905cab7d8450d9ee06d4e2463afa5 P 6a2f154bcc534134bd963d22d54ba278: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:03:43.486658 4511 ts_tablet_manager.cc:1929] T ec1905cab7d8450d9ee06d4e2463afa5 P 6a2f154bcc534134bd963d22d54ba278: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 14:03:43.486755 4511 log.cc:1199] T ec1905cab7d8450d9ee06d4e2463afa5 P 6a2f154bcc534134bd963d22d54ba278: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-0/wal/wals/ec1905cab7d8450d9ee06d4e2463afa5
I20260429 14:03:43.486955 4513 ts_tablet_manager.cc:1916] T ec1905cab7d8450d9ee06d4e2463afa5 P 52af864dbc164356a9ef9b9974c6be29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:03:43.487015 4511 ts_tablet_manager.cc:1950] T ec1905cab7d8450d9ee06d4e2463afa5 P 6a2f154bcc534134bd963d22d54ba278: Deleting consensus metadata
I20260429 14:03:43.487376 4512 ts_tablet_manager.cc:1929] T ec1905cab7d8450d9ee06d4e2463afa5 P 7de3fbd2cbcf477fa2b3dfc27674e490: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 14:03:43.487433 4512 log.cc:1199] T ec1905cab7d8450d9ee06d4e2463afa5 P 7de3fbd2cbcf477fa2b3dfc27674e490: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-1/wal/wals/ec1905cab7d8450d9ee06d4e2463afa5
I20260429 14:03:43.487403 3960 catalog_manager.cc:5002] TS 6a2f154bcc534134bd963d22d54ba278 (127.2.197.193:41417): tablet ec1905cab7d8450d9ee06d4e2463afa5 (table test-table [id=974ef469d98b43f9af5a7c93b1e50416]) successfully deleted
I20260429 14:03:43.487876 4512 ts_tablet_manager.cc:1950] T ec1905cab7d8450d9ee06d4e2463afa5 P 7de3fbd2cbcf477fa2b3dfc27674e490: Deleting consensus metadata
I20260429 14:03:43.488374 3962 catalog_manager.cc:5002] TS 7de3fbd2cbcf477fa2b3dfc27674e490 (127.2.197.194:40607): tablet ec1905cab7d8450d9ee06d4e2463afa5 (table test-table [id=974ef469d98b43f9af5a7c93b1e50416]) successfully deleted
I20260429 14:03:43.489122 4513 ts_tablet_manager.cc:1929] T ec1905cab7d8450d9ee06d4e2463afa5 P 52af864dbc164356a9ef9b9974c6be29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 14:03:43.489197 4513 log.cc:1199] T ec1905cab7d8450d9ee06d4e2463afa5 P 52af864dbc164356a9ef9b9974c6be29: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777471413505459-2839-0/minicluster-data/ts-2/wal/wals/ec1905cab7d8450d9ee06d4e2463afa5
I20260429 14:03:43.489421 4513 ts_tablet_manager.cc:1950] T ec1905cab7d8450d9ee06d4e2463afa5 P 52af864dbc164356a9ef9b9974c6be29: Deleting consensus metadata
I20260429 14:03:43.490000 3962 catalog_manager.cc:5002] TS 52af864dbc164356a9ef9b9974c6be29 (127.2.197.195:33361): tablet ec1905cab7d8450d9ee06d4e2463afa5 (table test-table [id=974ef469d98b43f9af5a7c93b1e50416]) successfully deleted
Apr 29 14:03:43 dist-test-slave-8n77 krb5kdc[3927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471422, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.197.193@KRBTEST.COM
I20260429 14:03:43.495743 4485 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:43.485092 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42726 (local address 127.2.197.193:41417)
0429 14:03:43.485136 (+ 44us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:43.485139 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:43.486586 (+ 1447us) server_negotiation.cc:408] Connection header received
0429 14:03:43.486679 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:43.486680 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:43.486691 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:43.486767 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:43.487808 (+ 1041us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:43.488420 (+ 612us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:43.489201 (+ 781us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:43.489587 (+ 386us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:43.492345 (+ 2758us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:43.492367 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:43.492368 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:43.492375 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:43.494521 (+ 2146us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:43.494989 (+ 468us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:43.494992 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:43.494993 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:43.495023 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:43.495187 (+ 164us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:43.495188 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:43.495189 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:43.495316 (+ 127us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:43.495478 (+ 162us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:43.495523 (+ 45us) server_negotiation.cc:300] Negotiation successful
0429 14:03:43.495625 (+ 102us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260429 14:03:43.496161 4174 server_base.cc:1143] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42726
I20260429 14:03:43.503721 4024 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:43.497335 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49812 (local address 127.2.197.254:38449)
0429 14:03:43.497412 (+ 77us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:43.497414 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:43.497423 (+ 9us) server_negotiation.cc:408] Connection header received
0429 14:03:43.497485 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:43.497487 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:43.497497 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:43.497553 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:43.498199 (+ 646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:43.498778 (+ 579us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:43.499406 (+ 628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:43.499622 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:43.500640 (+ 1018us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:43.500644 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:43.500644 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:43.500652 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:43.502535 (+ 1883us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:43.502950 (+ 415us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:43.502951 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:43.502951 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:43.502983 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:43.503147 (+ 164us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:43.503147 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:43.503148 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:43.503252 (+ 104us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:43.503364 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:43.503481 (+ 117us) server_negotiation.cc:300] Negotiation successful
0429 14:03:43.503609 (+ 128us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260429 14:03:43.504053 3975 server_base.cc:1143] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49812
I20260429 14:03:43.504678 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 4049
I20260429 14:03:43.511853 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 4191
I20260429 14:03:43.517877 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 4333
I20260429 14:03:43.524379 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 3943
2026-04-29T14:03:43Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3660 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:03:43 dist-test-slave-8n77 krb5kdc[4551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:03:43 dist-test-slave-8n77 krb5kdc[4551](info): set up 2 sockets
Apr 29 14:03:43 dist-test-slave-8n77 krb5kdc[4551](info): commencing operation
krb5kdc: starting...
W20260429 14:03:45.584208 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.037s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:03:45 dist-test-slave-8n77 krb5kdc[4551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:03:45Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:03:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:03:45.752975 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:36237
--webserver_interface=127.2.197.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:34603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:36237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:03:45.842732 4567 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:03:45.842979 4567 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:03:45.843006 4567 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:03:45.844782 4567 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:03:45.844846 4567 flags.cc:432] Enabled experimental flag: --txn_manager_enabled=true
W20260429 14:03:45.844859 4567 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:03:45.844873 4567 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:03:45.844887 4567 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:03:45.844899 4567 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:03:45.846798 4567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:34603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:36237
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:36237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:03:45.847075 4567 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:03:45.847301 4567 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:03:45.850019 4575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:03:45.850028 4573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:03:45.850150 4572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:03:45.850950 4567 server_base.cc:1061] running on GCE node
I20260429 14:03:45.851194 4567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:03:45.851467 4567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:03:45.852677 4567 hybrid_clock.cc:648] HybridClock initialized: now 1777471425852634 us; error 53 us; skew 500 ppm
Apr 29 14:03:45 dist-test-slave-8n77 krb5kdc[4551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471425, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:03:45.855409 4567 init.cc:377] Logged in from keytab as kudu/127.2.197.254@KRBTEST.COM (short username kudu)
I20260429 14:03:45.856031 4567 webserver.cc:492] Webserver started at http://127.2.197.254:40701/ using document root <none> and password file <none>
I20260429 14:03:45.856254 4567 fs_manager.cc:362] Metadata directory not provided
I20260429 14:03:45.856298 4567 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:03:45.856410 4567 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:03:45.857402 4567 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "1aeac3f977874eb79cf735db9587a3eb"
format_stamp: "Formatted at 2026-04-29 14:03:45 on dist-test-slave-8n77"
server_key: "4ff53e9b408df71ccb90be3ce3484510"
server_key_iv: "a0cb08eb1ba330a91ec1e88a2ea7e3e8"
server_key_version: "encryptionkey@0"
I20260429 14:03:45.857744 4567 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "1aeac3f977874eb79cf735db9587a3eb"
format_stamp: "Formatted at 2026-04-29 14:03:45 on dist-test-slave-8n77"
server_key: "4ff53e9b408df71ccb90be3ce3484510"
server_key_iv: "a0cb08eb1ba330a91ec1e88a2ea7e3e8"
server_key_version: "encryptionkey@0"
I20260429 14:03:45.859236 4567 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260429 14:03:45.860252 4582 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:45.860599 4567 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:03:45.860738 4567 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "1aeac3f977874eb79cf735db9587a3eb"
format_stamp: "Formatted at 2026-04-29 14:03:45 on dist-test-slave-8n77"
server_key: "4ff53e9b408df71ccb90be3ce3484510"
server_key_iv: "a0cb08eb1ba330a91ec1e88a2ea7e3e8"
server_key_version: "encryptionkey@0"
I20260429 14:03:45.860852 4567 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:03:45.884004 4567 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:03:45.888391 4567 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:03:45.888655 4567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:03:45.895159 4567 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:36237
I20260429 14:03:45.895231 4644 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:36237 every 8 connection(s)
I20260429 14:03:45.895552 4567 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:03:45.896395 4645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:03:45.899207 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 4567
I20260429 14:03:45.899295 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:03:45.899467 2839 external_mini_cluster.cc:1468] Setting key 65df14b16aa7dd36e1ba9416c9626f3a
I20260429 14:03:45.900621 4645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1aeac3f977874eb79cf735db9587a3eb: Bootstrap starting.
I20260429 14:03:45.901265 4645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1aeac3f977874eb79cf735db9587a3eb: Neither blocks nor log segments found. Creating new log.
I20260429 14:03:45.901517 4645 log.cc:826] T 00000000000000000000000000000000 P 1aeac3f977874eb79cf735db9587a3eb: Log is configured to *not* fsync() on all Append() calls
I20260429 14:03:45.902220 4645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1aeac3f977874eb79cf735db9587a3eb: No bootstrap required, opened a new log
I20260429 14:03:45.903407 4645 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1aeac3f977874eb79cf735db9587a3eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aeac3f977874eb79cf735db9587a3eb" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 36237 } }
I20260429 14:03:45.903671 4645 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1aeac3f977874eb79cf735db9587a3eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:03:45.903788 4645 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1aeac3f977874eb79cf735db9587a3eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aeac3f977874eb79cf735db9587a3eb, State: Initialized, Role: FOLLOWER
I20260429 14:03:45.904124 4645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1aeac3f977874eb79cf735db9587a3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aeac3f977874eb79cf735db9587a3eb" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 36237 } }
I20260429 14:03:45.904284 4645 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1aeac3f977874eb79cf735db9587a3eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:03:45.904357 4645 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1aeac3f977874eb79cf735db9587a3eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:03:45.904441 4645 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1aeac3f977874eb79cf735db9587a3eb [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:03:45.905819 4645 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1aeac3f977874eb79cf735db9587a3eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aeac3f977874eb79cf735db9587a3eb" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 36237 } }
I20260429 14:03:45.906019 4645 leader_election.cc:304] T 00000000000000000000000000000000 P 1aeac3f977874eb79cf735db9587a3eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1aeac3f977874eb79cf735db9587a3eb; no voters:
I20260429 14:03:45.906255 4645 leader_election.cc:290] T 00000000000000000000000000000000 P 1aeac3f977874eb79cf735db9587a3eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:03:45.906414 4650 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1aeac3f977874eb79cf735db9587a3eb [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:03:45.906592 4645 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1aeac3f977874eb79cf735db9587a3eb [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:03:45.906636 4650 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1aeac3f977874eb79cf735db9587a3eb [term 1 LEADER]: Becoming Leader. State: Replica: 1aeac3f977874eb79cf735db9587a3eb, State: Running, Role: LEADER
I20260429 14:03:45.906797 4650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1aeac3f977874eb79cf735db9587a3eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aeac3f977874eb79cf735db9587a3eb" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 36237 } }
Apr 29 14:03:45 dist-test-slave-8n77 krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:03:45.907160 4651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1aeac3f977874eb79cf735db9587a3eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1aeac3f977874eb79cf735db9587a3eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aeac3f977874eb79cf735db9587a3eb" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 36237 } } }
I20260429 14:03:45.907241 4651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1aeac3f977874eb79cf735db9587a3eb [sys.catalog]: This master's current role is: LEADER
I20260429 14:03:45.907521 4652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1aeac3f977874eb79cf735db9587a3eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1aeac3f977874eb79cf735db9587a3eb. Latest consensus state: current_term: 1 leader_uuid: "1aeac3f977874eb79cf735db9587a3eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aeac3f977874eb79cf735db9587a3eb" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 36237 } } }
I20260429 14:03:45.907656 4652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1aeac3f977874eb79cf735db9587a3eb [sys.catalog]: This master's current role is: LEADER
I20260429 14:03:45.907984 4654 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:03:45.908514 4654 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:03:45.910720 4654 catalog_manager.cc:1357] Generated new cluster ID: 86a6e167d9a945f891ca69781383bac5
I20260429 14:03:45.910795 4654 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:03:45.913755 4648 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:45.900146 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51936 (local address 127.2.197.254:36237)
0429 14:03:45.900367 (+ 221us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:45.900377 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:45.900398 (+ 21us) server_negotiation.cc:408] Connection header received
0429 14:03:45.900876 (+ 478us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:45.900883 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:45.900928 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:45.901064 (+ 136us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:45.902433 (+ 1369us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:45.903293 (+ 860us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:45.904130 (+ 837us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:45.904440 (+ 310us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:45.907626 (+ 3186us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:45.907637 (+ 11us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:45.907637 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:45.907653 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:45.912301 (+ 4648us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:45.912719 (+ 418us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:45.912724 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:45.912725 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:45.912773 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:45.912968 (+ 195us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:45.912969 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:45.912969 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:45.913182 (+ 213us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:45.913341 (+ 159us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:45.913465 (+ 124us) server_negotiation.cc:300] Negotiation successful
0429 14:03:45.913603 (+ 138us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":96,"threads_started":1}
I20260429 14:03:45.921819 4654 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:03:45.922557 4654 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:03:45.927716 4654 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 1aeac3f977874eb79cf735db9587a3eb: Generated new TSK 0
I20260429 14:03:45.927981 4654 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:03:45.991222 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.193:0
--local_ip_for_outbound_sockets=127.2.197.193
--webserver_interface=127.2.197.193
--webserver_port=0
--tserver_master_addrs=127.2.197.254:36237
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:34603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:03:46.077657 4673 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:03:46.077917 4673 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:03:46.077965 4673 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:03:46.080026 4673 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:03:46.080145 4673 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:03:46.080186 4673 flags.cc:432] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260429 14:03:46.080214 4673 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260429 14:03:46.080256 4673 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.193
I20260429 14:03:46.082842 4673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:34603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.197.193
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.197.254:36237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.193
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:03:46.083160 4673 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:03:46.083364 4673 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:03:46.086457 4681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:03:46.086422 4679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:03:46.086980 4673 server_base.cc:1061] running on GCE node
W20260429 14:03:46.086557 4678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:03:46.087303 4673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:03:46.087564 4673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:03:46.088738 4673 hybrid_clock.cc:648] HybridClock initialized: now 1777471426088716 us; error 40 us; skew 500 ppm
Apr 29 14:03:46 dist-test-slave-8n77 krb5kdc[4551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471426, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:03:46.091485 4673 init.cc:377] Logged in from keytab as kudu/127.2.197.193@KRBTEST.COM (short username kudu)
I20260429 14:03:46.092069 4673 webserver.cc:492] Webserver started at http://127.2.197.193:45429/ using document root <none> and password file <none>
I20260429 14:03:46.092293 4673 fs_manager.cc:362] Metadata directory not provided
I20260429 14:03:46.092344 4673 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:03:46.092440 4673 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:03:46.093473 4673 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-0/data/instance:
uuid: "ecdb77fe301541a4a0f652e78308dc80"
format_stamp: "Formatted at 2026-04-29 14:03:46 on dist-test-slave-8n77"
server_key: "f3ce293d631b84e3722313e1cc743c5d"
server_key_iv: "e247dcc8b05d8b5a9ce3cd65cce7c0f6"
server_key_version: "encryptionkey@0"
I20260429 14:03:46.093817 4673 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance:
uuid: "ecdb77fe301541a4a0f652e78308dc80"
format_stamp: "Formatted at 2026-04-29 14:03:46 on dist-test-slave-8n77"
server_key: "f3ce293d631b84e3722313e1cc743c5d"
server_key_iv: "e247dcc8b05d8b5a9ce3cd65cce7c0f6"
server_key_version: "encryptionkey@0"
I20260429 14:03:46.095108 4673 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260429 14:03:46.095850 4688 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:46.096035 4673 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:03:46.096153 4673 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-0/wal
uuid: "ecdb77fe301541a4a0f652e78308dc80"
format_stamp: "Formatted at 2026-04-29 14:03:46 on dist-test-slave-8n77"
server_key: "f3ce293d631b84e3722313e1cc743c5d"
server_key_iv: "e247dcc8b05d8b5a9ce3cd65cce7c0f6"
server_key_version: "encryptionkey@0"
I20260429 14:03:46.096228 4673 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:03:46.112531 4673 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:03:46.114532 4673 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:03:46.114678 4673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:03:46.115274 4673 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:03:46.115319 4673 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:46.115351 4673 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:03:46.115382 4673 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 29 14:03:46 dist-test-slave-8n77 krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471426, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:03:46.123790 4673 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.193:42969
I20260429 14:03:46.123972 4803 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.193:42969 every 8 connection(s)
I20260429 14:03:46.124286 4673 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
I20260429 14:03:46.125252 4648 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:46.115785 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:57597 (local address 127.2.197.254:36237)
0429 14:03:46.115859 (+ 74us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:46.115862 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:46.116010 (+ 148us) server_negotiation.cc:408] Connection header received
0429 14:03:46.116518 (+ 508us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:46.116519 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:46.116532 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:46.116586 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:46.117703 (+ 1117us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.118619 (+ 916us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:46.119245 (+ 626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.119427 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:46.121698 (+ 2271us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:46.121710 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:46.121711 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:46.121718 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:46.123636 (+ 1918us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:46.124170 (+ 534us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:46.124172 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:46.124172 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:46.124200 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:46.124518 (+ 318us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:46.124519 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:46.124519 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:46.124708 (+ 189us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:46.124782 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:46.125093 (+ 311us) server_negotiation.cc:300] Negotiation successful
0429 14:03:46.125189 (+ 96us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260429 14:03:46.125326 4697 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:46.115712 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:36237 (local address 127.2.197.193:57597)
0429 14:03:46.115894 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:46.115925 (+ 31us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:46.116392 (+ 467us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:46.116708 (+ 316us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:46.116711 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:46.117159 (+ 448us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:03:46.117560 (+ 401us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:46.117564 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.118719 (+ 1155us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:46.118720 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:46.119116 (+ 396us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:46.119118 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.119284 (+ 166us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:46.120059 (+ 775us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:03:46.120097 (+ 38us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:03:46.121583 (+ 1486us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:03:46.123781 (+ 2198us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:46.123787 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:46.123789 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:46.124085 (+ 296us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:46.124259 (+ 174us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:46.124260 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:46.124260 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:46.124332 (+ 72us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:46.124811 (+ 479us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:03:46.124816 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:03:46.124958 (+ 142us) client_negotiation.cc:770] Sending connection context
0429 14:03:46.125046 (+ 88us) client_negotiation.cc:241] Negotiation successful
0429 14:03:46.125158 (+ 112us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":102,"threads_started":1}
I20260429 14:03:46.125870 4804 heartbeater.cc:344] Connected to a master server at 127.2.197.254:36237
I20260429 14:03:46.125950 4804 heartbeater.cc:461] Registering TS with master...
I20260429 14:03:46.126152 4804 heartbeater.cc:507] Master 127.2.197.254:36237 requested a full tablet report, sending...
I20260429 14:03:46.126739 4598 ts_manager.cc:194] Registered new tserver with Master: ecdb77fe301541a4a0f652e78308dc80 (127.2.197.193:42969)
I20260429 14:03:46.127395 4598 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.193@KRBTEST.COM'} at 127.2.197.193:57597
I20260429 14:03:46.127732 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 4673
I20260429 14:03:46.127879 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance
I20260429 14:03:46.128063 2839 external_mini_cluster.cc:1468] Setting key d9e403174931aec9580939cbe65e1677
I20260429 14:03:46.134084 4648 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:46.127491 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:46625 (local address 127.2.197.254:36237)
0429 14:03:46.127575 (+ 84us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:46.127577 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:46.127707 (+ 130us) server_negotiation.cc:408] Connection header received
0429 14:03:46.127840 (+ 133us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:46.127841 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:46.127857 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:46.127921 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:46.128477 (+ 556us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.129050 (+ 573us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:46.129835 (+ 785us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.129961 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:46.130770 (+ 809us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:46.130776 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:46.130776 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:46.130786 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:46.132479 (+ 1693us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:46.132971 (+ 492us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:46.132973 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:46.132973 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:46.133008 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:46.133183 (+ 175us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:46.133184 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:46.133185 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:46.133315 (+ 130us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:46.133541 (+ 226us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:46.133754 (+ 213us) server_negotiation.cc:300] Negotiation successful
0429 14:03:46.133979 (+ 225us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260429 14:03:46.134101 4810 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:46.127429 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:36237 (local address 127.2.197.193:46625)
0429 14:03:46.127640 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:46.127653 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:46.127724 (+ 71us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:46.127976 (+ 252us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:46.127977 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:46.128156 (+ 179us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:03:46.128387 (+ 231us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:46.128390 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.129128 (+ 738us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:46.129129 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:46.129743 (+ 614us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:46.129746 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.129823 (+ 77us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:46.130368 (+ 545us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:03:46.130374 (+ 6us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:03:46.130666 (+ 292us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:03:46.132663 (+ 1997us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:46.132665 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:46.132666 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:46.132874 (+ 208us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:46.133084 (+ 210us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:46.133084 (+ 0us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:46.133085 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:46.133123 (+ 38us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:46.133528 (+ 405us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:03:46.133530 (+ 2us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:03:46.133607 (+ 77us) client_negotiation.cc:770] Sending connection context
0429 14:03:46.133723 (+ 116us) client_negotiation.cc:241] Negotiation successful
0429 14:03:46.133980 (+ 257us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":87,"threads_started":1}
WARNING: no policy specified for kudu/127.2.197.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:03:46.190009 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.194:0
--local_ip_for_outbound_sockets=127.2.197.194
--webserver_interface=127.2.197.194
--webserver_port=0
--tserver_master_addrs=127.2.197.254:36237
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:34603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:03:46.275321 4815 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:03:46.275534 4815 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:03:46.275573 4815 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:03:46.277352 4815 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:03:46.277421 4815 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:03:46.277446 4815 flags.cc:432] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260429 14:03:46.277458 4815 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260429 14:03:46.277477 4815 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.194
I20260429 14:03:46.279452 4815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:34603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.197.194
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.197.254:36237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.194
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:03:46.279714 4815 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:03:46.279913 4815 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:03:46.282876 4823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:03:46.282913 4821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:03:46.282929 4820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:03:46.283035 4815 server_base.cc:1061] running on GCE node
I20260429 14:03:46.283303 4815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:03:46.283555 4815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:03:46.284739 4815 hybrid_clock.cc:648] HybridClock initialized: now 1777471426284707 us; error 38 us; skew 500 ppm
Apr 29 14:03:46 dist-test-slave-8n77 krb5kdc[4551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471426, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:03:46.287420 4815 init.cc:377] Logged in from keytab as kudu/127.2.197.194@KRBTEST.COM (short username kudu)
I20260429 14:03:46.288066 4815 webserver.cc:492] Webserver started at http://127.2.197.194:40081/ using document root <none> and password file <none>
I20260429 14:03:46.288278 4815 fs_manager.cc:362] Metadata directory not provided
I20260429 14:03:46.288341 4815 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:03:46.288451 4815 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:03:46.289440 4815 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-1/data/instance:
uuid: "f50d561a4167470b961a3f7dbeb6ef3e"
format_stamp: "Formatted at 2026-04-29 14:03:46 on dist-test-slave-8n77"
server_key: "5ff96531263310b7f5789f64839ce06c"
server_key_iv: "bf19508a51f6d41d359022d10f995df2"
server_key_version: "encryptionkey@0"
I20260429 14:03:46.289799 4815 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance:
uuid: "f50d561a4167470b961a3f7dbeb6ef3e"
format_stamp: "Formatted at 2026-04-29 14:03:46 on dist-test-slave-8n77"
server_key: "5ff96531263310b7f5789f64839ce06c"
server_key_iv: "bf19508a51f6d41d359022d10f995df2"
server_key_version: "encryptionkey@0"
I20260429 14:03:46.291194 4815 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260429 14:03:46.292013 4830 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:46.292339 4815 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:03:46.292521 4815 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-1/wal
uuid: "f50d561a4167470b961a3f7dbeb6ef3e"
format_stamp: "Formatted at 2026-04-29 14:03:46 on dist-test-slave-8n77"
server_key: "5ff96531263310b7f5789f64839ce06c"
server_key_iv: "bf19508a51f6d41d359022d10f995df2"
server_key_version: "encryptionkey@0"
I20260429 14:03:46.292680 4815 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:03:46.302126 4815 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:03:46.304250 4815 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:03:46.304415 4815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:03:46.305150 4815 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:03:46.305213 4815 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:46.305259 4815 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:03:46.305289 4815 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 29 14:03:46 dist-test-slave-8n77 krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471426, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:03:46.313340 4815 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.194:37203
I20260429 14:03:46.313423 4945 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.194:37203 every 8 connection(s)
I20260429 14:03:46.313969 4815 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
I20260429 14:03:46.315194 4648 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:46.305681 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:34627 (local address 127.2.197.254:36237)
0429 14:03:46.305765 (+ 84us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:46.305768 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:46.305963 (+ 195us) server_negotiation.cc:408] Connection header received
0429 14:03:46.306540 (+ 577us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:46.306542 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:46.306555 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:46.306604 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:46.307308 (+ 704us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.307945 (+ 637us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:46.308600 (+ 655us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.308770 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:46.311270 (+ 2500us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:46.311276 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:46.311276 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:46.311286 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:46.313632 (+ 2346us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:46.314093 (+ 461us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:46.314094 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:46.314095 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:46.314122 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:46.314418 (+ 296us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:46.314419 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:46.314419 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:46.314556 (+ 137us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:46.314652 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:46.314998 (+ 346us) server_negotiation.cc:300] Negotiation successful
0429 14:03:46.315101 (+ 103us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260429 14:03:46.315316 4839 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:46.305607 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:36237 (local address 127.2.197.194:34627)
0429 14:03:46.305829 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:46.305864 (+ 35us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:46.306439 (+ 575us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:46.306654 (+ 215us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:46.306658 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:46.306943 (+ 285us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:03:46.307199 (+ 256us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:46.307202 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.308056 (+ 854us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:46.308058 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:46.308473 (+ 415us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:46.308475 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.308648 (+ 173us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:46.309306 (+ 658us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:03:46.309335 (+ 29us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:03:46.311065 (+ 1730us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:03:46.313758 (+ 2693us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:46.313763 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:46.313765 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:46.314005 (+ 240us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:46.314199 (+ 194us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:46.314200 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:46.314200 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:46.314339 (+ 139us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:46.314641 (+ 302us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:03:46.314646 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:03:46.314844 (+ 198us) client_negotiation.cc:770] Sending connection context
0429 14:03:46.314968 (+ 124us) client_negotiation.cc:241] Negotiation successful
0429 14:03:46.315117 (+ 149us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":111,"threads_started":1}
I20260429 14:03:46.315344 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 4815
I20260429 14:03:46.315456 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance
I20260429 14:03:46.315616 2839 external_mini_cluster.cc:1468] Setting key 75d34f1b0c193a9ddf52b54ea9b6ca46
I20260429 14:03:46.316274 4946 heartbeater.cc:344] Connected to a master server at 127.2.197.254:36237
I20260429 14:03:46.316366 4946 heartbeater.cc:461] Registering TS with master...
I20260429 14:03:46.316591 4946 heartbeater.cc:507] Master 127.2.197.254:36237 requested a full tablet report, sending...
I20260429 14:03:46.317328 4599 ts_manager.cc:194] Registered new tserver with Master: f50d561a4167470b961a3f7dbeb6ef3e (127.2.197.194:37203)
I20260429 14:03:46.317873 4599 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.194@KRBTEST.COM'} at 127.2.197.194:34627
I20260429 14:03:46.323865 4648 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:46.317179 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:40931 (local address 127.2.197.254:36237)
0429 14:03:46.317263 (+ 84us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:46.317265 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:46.317272 (+ 7us) server_negotiation.cc:408] Connection header received
0429 14:03:46.317300 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:46.317301 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:46.317311 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:46.317356 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:46.317784 (+ 428us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.318580 (+ 796us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:46.319392 (+ 812us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.319537 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:46.320698 (+ 1161us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:46.320706 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:46.320706 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:46.320718 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:46.322710 (+ 1992us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:46.323114 (+ 404us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:46.323115 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:46.323115 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:46.323138 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:46.323313 (+ 175us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:46.323314 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:46.323314 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:46.323428 (+ 114us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:46.323479 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:46.323717 (+ 238us) server_negotiation.cc:300] Negotiation successful
0429 14:03:46.323798 (+ 81us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260429 14:03:46.323927 4952 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:46.317029 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:36237 (local address 127.2.197.194:40931)
0429 14:03:46.317177 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:46.317189 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:46.317248 (+ 59us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:46.317358 (+ 110us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:46.317359 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:46.317527 (+ 168us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:03:46.317691 (+ 164us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:46.317692 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.318676 (+ 984us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:46.318678 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:46.319287 (+ 609us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:46.319290 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.319371 (+ 81us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:46.320029 (+ 658us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:03:46.320035 (+ 6us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:03:46.320493 (+ 458us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:03:46.322810 (+ 2317us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:46.322812 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:46.322813 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:46.323031 (+ 218us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:46.323209 (+ 178us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:46.323210 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:46.323210 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:46.323250 (+ 40us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:46.323496 (+ 246us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:03:46.323497 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:03:46.323563 (+ 66us) client_negotiation.cc:770] Sending connection context
0429 14:03:46.323644 (+ 81us) client_negotiation.cc:241] Negotiation successful
0429 14:03:46.323781 (+ 137us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for kudu/127.2.197.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:03:46.370131 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.195:0
--local_ip_for_outbound_sockets=127.2.197.195
--webserver_interface=127.2.197.195
--webserver_port=0
--tserver_master_addrs=127.2.197.254:36237
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:34603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:03:46.451656 4957 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:03:46.451862 4957 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:03:46.451891 4957 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:03:46.453887 4957 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:03:46.453955 4957 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:03:46.453980 4957 flags.cc:432] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260429 14:03:46.453992 4957 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260429 14:03:46.454010 4957 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.195
I20260429 14:03:46.455979 4957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:34603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.197.195
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.197.254:36237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.195
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:03:46.456251 4957 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:03:46.456465 4957 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:03:46.459044 4962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:03:46.459090 4957 server_base.cc:1061] running on GCE node
W20260429 14:03:46.459036 4963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:03:46.459033 4965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:03:46.459340 4957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:03:46.459540 4957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:03:46.460690 4957 hybrid_clock.cc:648] HybridClock initialized: now 1777471426460677 us; error 27 us; skew 500 ppm
Apr 29 14:03:46 dist-test-slave-8n77 krb5kdc[4551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471426, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:03:46.463359 4957 init.cc:377] Logged in from keytab as kudu/127.2.197.195@KRBTEST.COM (short username kudu)
I20260429 14:03:46.463949 4957 webserver.cc:492] Webserver started at http://127.2.197.195:37253/ using document root <none> and password file <none>
I20260429 14:03:46.464165 4957 fs_manager.cc:362] Metadata directory not provided
I20260429 14:03:46.464226 4957 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:03:46.464334 4957 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:03:46.465286 4957 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-2/data/instance:
uuid: "12d3554b22244952a8cb77fa76ad9e38"
format_stamp: "Formatted at 2026-04-29 14:03:46 on dist-test-slave-8n77"
server_key: "ece4eb0cc4343a82baaba4d53b0937e8"
server_key_iv: "655230e6d5612c1b12da40d062db6a4b"
server_key_version: "encryptionkey@0"
I20260429 14:03:46.465632 4957 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance:
uuid: "12d3554b22244952a8cb77fa76ad9e38"
format_stamp: "Formatted at 2026-04-29 14:03:46 on dist-test-slave-8n77"
server_key: "ece4eb0cc4343a82baaba4d53b0937e8"
server_key_iv: "655230e6d5612c1b12da40d062db6a4b"
server_key_version: "encryptionkey@0"
I20260429 14:03:46.466987 4957 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260429 14:03:46.467736 4972 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:46.467937 4957 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:03:46.468103 4957 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-2/wal
uuid: "12d3554b22244952a8cb77fa76ad9e38"
format_stamp: "Formatted at 2026-04-29 14:03:46 on dist-test-slave-8n77"
server_key: "ece4eb0cc4343a82baaba4d53b0937e8"
server_key_iv: "655230e6d5612c1b12da40d062db6a4b"
server_key_version: "encryptionkey@0"
I20260429 14:03:46.468230 4957 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:03:46.486407 4957 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:03:46.488499 4957 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:03:46.488624 4957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:03:46.489090 4957 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:03:46.489135 4957 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:46.489159 4957 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:03:46.489172 4957 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 29 14:03:46 dist-test-slave-8n77 krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471426, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:03:46.496670 4957 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.195:40127
I20260429 14:03:46.496721 5087 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.195:40127 every 8 connection(s)
I20260429 14:03:46.497184 4957 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
I20260429 14:03:46.499373 4648 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:46.489621 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:53393 (local address 127.2.197.254:36237)
0429 14:03:46.489697 (+ 76us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:46.489700 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:46.489886 (+ 186us) server_negotiation.cc:408] Connection header received
0429 14:03:46.490400 (+ 514us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:46.490402 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:46.490418 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:46.490512 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:46.491498 (+ 986us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.492337 (+ 839us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:46.493362 (+ 1025us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.493591 (+ 229us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:46.496276 (+ 2685us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:46.496281 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:46.496281 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:46.496292 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:46.498130 (+ 1838us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:46.498602 (+ 472us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:46.498603 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:46.498604 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:46.498627 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:46.498817 (+ 190us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:46.498818 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:46.498818 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:46.498914 (+ 96us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:46.498964 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:46.499234 (+ 270us) server_negotiation.cc:300] Negotiation successful
0429 14:03:46.499311 (+ 77us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260429 14:03:46.499402 4981 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:46.489576 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:36237 (local address 127.2.197.195:53393)
0429 14:03:46.489759 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:46.489788 (+ 29us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:46.490253 (+ 465us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:46.490550 (+ 297us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:46.490554 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:46.490994 (+ 440us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:03:46.491386 (+ 392us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:46.491391 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.492441 (+ 1050us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:46.492442 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:46.493244 (+ 802us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:46.493247 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.493388 (+ 141us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:46.493991 (+ 603us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:03:46.494003 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:03:46.496159 (+ 2156us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:03:46.498224 (+ 2065us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:46.498229 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:46.498231 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:46.498511 (+ 280us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:46.498696 (+ 185us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:46.498696 (+ 0us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:46.498697 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:46.498752 (+ 55us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:46.498966 (+ 214us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:03:46.498969 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:03:46.499070 (+ 101us) client_negotiation.cc:770] Sending connection context
0429 14:03:46.499152 (+ 82us) client_negotiation.cc:241] Negotiation successful
0429 14:03:46.499259 (+ 107us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":115,"threads_started":1}
I20260429 14:03:46.499994 5088 heartbeater.cc:344] Connected to a master server at 127.2.197.254:36237
I20260429 14:03:46.500082 5088 heartbeater.cc:461] Registering TS with master...
I20260429 14:03:46.500288 5088 heartbeater.cc:507] Master 127.2.197.254:36237 requested a full tablet report, sending...
I20260429 14:03:46.500982 4598 ts_manager.cc:194] Registered new tserver with Master: 12d3554b22244952a8cb77fa76ad9e38 (127.2.197.195:40127)
I20260429 14:03:46.501503 4598 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.195@KRBTEST.COM'} at 127.2.197.195:53393
I20260429 14:03:46.505426 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 4957
I20260429 14:03:46.505533 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance
I20260429 14:03:46.505683 2839 external_mini_cluster.cc:1468] Setting key c6cec126ee1e10a890818eff11231dc2
I20260429 14:03:46.506852 2839 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260429 14:03:46.507145 4648 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:46.501375 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:50575 (local address 127.2.197.254:36237)
0429 14:03:46.501418 (+ 43us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:46.501420 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:46.501559 (+ 139us) server_negotiation.cc:408] Connection header received
0429 14:03:46.501643 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:46.501644 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:46.501656 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:46.501687 (+ 31us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:46.502161 (+ 474us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.502692 (+ 531us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:46.503329 (+ 637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.503524 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:46.504158 (+ 634us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:46.504161 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:46.504162 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:46.504168 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:46.505956 (+ 1788us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:46.506363 (+ 407us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:46.506365 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:46.506365 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:46.506393 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:46.506569 (+ 176us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:46.506570 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:46.506570 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:46.506685 (+ 115us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:46.506763 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:46.507007 (+ 244us) server_negotiation.cc:300] Negotiation successful
0429 14:03:46.507085 (+ 78us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260429 14:03:46.507172 5094 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:46.501304 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:36237 (local address 127.2.197.195:50575)
0429 14:03:46.501477 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:46.501488 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:46.501576 (+ 88us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:46.501770 (+ 194us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:46.501771 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:46.501916 (+ 145us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:03:46.502079 (+ 163us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:46.502081 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.502847 (+ 766us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:46.502848 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:46.503235 (+ 387us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:46.503236 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.503321 (+ 85us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:46.503812 (+ 491us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:03:46.503816 (+ 4us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:03:46.504064 (+ 248us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:03:46.506057 (+ 1993us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:46.506058 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:46.506058 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:46.506233 (+ 175us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:46.506461 (+ 228us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:46.506462 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:46.506462 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:46.506499 (+ 37us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:46.506801 (+ 302us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:03:46.506803 (+ 2us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:03:46.506873 (+ 70us) client_negotiation.cc:770] Sending connection context
0429 14:03:46.506954 (+ 81us) client_negotiation.cc:241] Negotiation successful
0429 14:03:46.507060 (+ 106us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":76,"threads_started":1}
Apr 29 14:03:46 dist-test-slave-8n77 krb5kdc[4551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Apr 29 14:03:46 dist-test-slave-8n77 krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:03:46.530478 4648 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:46.523160 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51952 (local address 127.2.197.254:36237)
0429 14:03:46.523235 (+ 75us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:46.523238 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:46.523246 (+ 8us) server_negotiation.cc:408] Connection header received
0429 14:03:46.523270 (+ 24us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:46.523271 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:46.523280 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:46.523331 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:46.524152 (+ 821us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.524675 (+ 523us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:46.525270 (+ 595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.525387 (+ 117us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:46.527821 (+ 2434us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:46.527826 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:46.527827 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:46.527834 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:46.529200 (+ 1366us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:46.529552 (+ 352us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:46.529553 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:46.529554 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:46.529580 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:46.529810 (+ 230us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:46.529811 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:46.529812 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:46.529911 (+ 99us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:46.530008 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:46.530178 (+ 170us) server_negotiation.cc:300] Negotiation successful
0429 14:03:46.530250 (+ 72us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260429 14:03:46.532114 4598 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 14:03:46.533408 4598 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 14:03:46.539971 5106 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:46.536466 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:40127 (local address 127.0.0.1:49430)
0429 14:03:46.536684 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:46.536717 (+ 33us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:46.536791 (+ 74us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:46.537014 (+ 223us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:46.537017 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:46.537020 (+ 3us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:03:46.537314 (+ 294us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:46.537316 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.538656 (+ 1340us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:46.538657 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:46.539658 (+ 1001us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:46.539660 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.539731 (+ 71us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:46.539778 (+ 47us) client_negotiation.cc:770] Sending connection context
0429 14:03:46.539853 (+ 75us) client_negotiation.cc:241] Negotiation successful
0429 14:03:46.539880 (+ 27us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":60,"threads_started":1}
I20260429 14:03:46.540153 5109 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:46.536716 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:37203 (local address 127.0.0.1:45904)
0429 14:03:46.536894 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:46.537104 (+ 210us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:46.537178 (+ 74us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:46.537431 (+ 253us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:46.537432 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:46.537433 (+ 1us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:03:46.537830 (+ 397us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:46.537832 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.539065 (+ 1233us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:46.539066 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:46.539826 (+ 760us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:46.539829 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.539900 (+ 71us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:46.539903 (+ 3us) client_negotiation.cc:770] Sending connection context
0429 14:03:46.539935 (+ 32us) client_negotiation.cc:241] Negotiation successful
0429 14:03:46.539973 (+ 38us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":70,"threads_started":1}
I20260429 14:03:46.540510 5108 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:46.536675 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49430 (local address 127.2.197.195:40127)
0429 14:03:46.536832 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:46.536839 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:46.536856 (+ 17us) server_negotiation.cc:408] Connection header received
0429 14:03:46.536896 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:46.536898 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:46.536916 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:46.536992 (+ 76us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:03:46.537406 (+ 414us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.538567 (+ 1161us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:46.539774 (+ 1207us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.540313 (+ 539us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:46.540366 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:46.540397 (+ 31us) server_negotiation.cc:300] Negotiation successful
0429 14:03:46.540435 (+ 38us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
I20260429 14:03:46.540685 5110 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:46.536988 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:45904 (local address 127.2.197.194:37203)
0429 14:03:46.537276 (+ 288us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:46.537282 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:46.537291 (+ 9us) server_negotiation.cc:408] Connection header received
0429 14:03:46.537326 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:46.537328 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:46.537344 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:46.537911 (+ 567us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:03:46.537998 (+ 87us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.538949 (+ 951us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:46.540027 (+ 1078us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.540506 (+ 479us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:46.540543 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:46.540572 (+ 29us) server_negotiation.cc:300] Negotiation successful
0429 14:03:46.540621 (+ 49us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":84,"threads_started":1}
I20260429 14:03:46.541119 5022 tablet_service.cc:1511] Processing CreateTablet for tablet dbdf08fc3dde4422aa60da02d73b3116 (DEFAULT_TABLE table=test-table [id=4d0f06ca6b0e48878392699d4106439a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:03:46.541283 5107 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:46.536568 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:52900 (local address 127.2.197.193:42969)
0429 14:03:46.536762 (+ 194us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:46.536769 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:46.536913 (+ 144us) server_negotiation.cc:408] Connection header received
0429 14:03:46.537035 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:46.537037 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:46.537050 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:46.537186 (+ 136us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:03:46.537710 (+ 524us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.538556 (+ 846us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:46.539937 (+ 1381us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.541108 (+ 1171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:46.541148 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:46.541179 (+ 31us) server_negotiation.cc:300] Negotiation successful
0429 14:03:46.541219 (+ 40us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":110,"threads_started":1}
I20260429 14:03:46.541201 4880 tablet_service.cc:1511] Processing CreateTablet for tablet dbdf08fc3dde4422aa60da02d73b3116 (DEFAULT_TABLE table=test-table [id=4d0f06ca6b0e48878392699d4106439a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:03:46.541431 5022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbdf08fc3dde4422aa60da02d73b3116. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:03:46.541462 5105 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:46.536479 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:42969 (local address 127.0.0.1:52900)
0429 14:03:46.536812 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:46.536848 (+ 36us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:46.536943 (+ 95us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:46.537315 (+ 372us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:46.537315 (+ 0us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:46.537317 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:03:46.537523 (+ 206us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:46.537525 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.538693 (+ 1168us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:46.538694 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:46.539661 (+ 967us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:46.539663 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.539733 (+ 70us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:46.539778 (+ 45us) client_negotiation.cc:770] Sending connection context
0429 14:03:46.539831 (+ 53us) client_negotiation.cc:241] Negotiation successful
0429 14:03:46.539875 (+ 44us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":113,"threads_started":1}
I20260429 14:03:46.541460 4880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbdf08fc3dde4422aa60da02d73b3116. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:03:46.541934 4738 tablet_service.cc:1511] Processing CreateTablet for tablet dbdf08fc3dde4422aa60da02d73b3116 (DEFAULT_TABLE table=test-table [id=4d0f06ca6b0e48878392699d4106439a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:03:46.542173 4738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbdf08fc3dde4422aa60da02d73b3116. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:03:46.543782 5111 tablet_bootstrap.cc:492] T dbdf08fc3dde4422aa60da02d73b3116 P f50d561a4167470b961a3f7dbeb6ef3e: Bootstrap starting.
I20260429 14:03:46.544255 5112 tablet_bootstrap.cc:492] T dbdf08fc3dde4422aa60da02d73b3116 P 12d3554b22244952a8cb77fa76ad9e38: Bootstrap starting.
I20260429 14:03:46.544872 5111 tablet_bootstrap.cc:654] T dbdf08fc3dde4422aa60da02d73b3116 P f50d561a4167470b961a3f7dbeb6ef3e: Neither blocks nor log segments found. Creating new log.
I20260429 14:03:46.544893 5113 tablet_bootstrap.cc:492] T dbdf08fc3dde4422aa60da02d73b3116 P ecdb77fe301541a4a0f652e78308dc80: Bootstrap starting.
I20260429 14:03:46.544998 5112 tablet_bootstrap.cc:654] T dbdf08fc3dde4422aa60da02d73b3116 P 12d3554b22244952a8cb77fa76ad9e38: Neither blocks nor log segments found. Creating new log.
I20260429 14:03:46.545420 5113 tablet_bootstrap.cc:654] T dbdf08fc3dde4422aa60da02d73b3116 P ecdb77fe301541a4a0f652e78308dc80: Neither blocks nor log segments found. Creating new log.
I20260429 14:03:46.545758 5112 log.cc:826] T dbdf08fc3dde4422aa60da02d73b3116 P 12d3554b22244952a8cb77fa76ad9e38: Log is configured to *not* fsync() on all Append() calls
I20260429 14:03:46.545784 5111 log.cc:826] T dbdf08fc3dde4422aa60da02d73b3116 P f50d561a4167470b961a3f7dbeb6ef3e: Log is configured to *not* fsync() on all Append() calls
I20260429 14:03:46.546029 5113 log.cc:826] T dbdf08fc3dde4422aa60da02d73b3116 P ecdb77fe301541a4a0f652e78308dc80: Log is configured to *not* fsync() on all Append() calls
I20260429 14:03:46.546566 5112 tablet_bootstrap.cc:492] T dbdf08fc3dde4422aa60da02d73b3116 P 12d3554b22244952a8cb77fa76ad9e38: No bootstrap required, opened a new log
I20260429 14:03:46.546653 5112 ts_tablet_manager.cc:1403] T dbdf08fc3dde4422aa60da02d73b3116 P 12d3554b22244952a8cb77fa76ad9e38: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260429 14:03:46.546661 5111 tablet_bootstrap.cc:492] T dbdf08fc3dde4422aa60da02d73b3116 P f50d561a4167470b961a3f7dbeb6ef3e: No bootstrap required, opened a new log
I20260429 14:03:46.546716 5111 ts_tablet_manager.cc:1403] T dbdf08fc3dde4422aa60da02d73b3116 P f50d561a4167470b961a3f7dbeb6ef3e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260429 14:03:46.546770 5113 tablet_bootstrap.cc:492] T dbdf08fc3dde4422aa60da02d73b3116 P ecdb77fe301541a4a0f652e78308dc80: No bootstrap required, opened a new log
I20260429 14:03:46.546844 5113 ts_tablet_manager.cc:1403] T dbdf08fc3dde4422aa60da02d73b3116 P ecdb77fe301541a4a0f652e78308dc80: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260429 14:03:46.548084 5112 raft_consensus.cc:359] T dbdf08fc3dde4422aa60da02d73b3116 P 12d3554b22244952a8cb77fa76ad9e38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:46.548237 5112 raft_consensus.cc:385] T dbdf08fc3dde4422aa60da02d73b3116 P 12d3554b22244952a8cb77fa76ad9e38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:03:46.548274 5112 raft_consensus.cc:740] T dbdf08fc3dde4422aa60da02d73b3116 P 12d3554b22244952a8cb77fa76ad9e38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12d3554b22244952a8cb77fa76ad9e38, State: Initialized, Role: FOLLOWER
I20260429 14:03:46.548393 5112 consensus_queue.cc:260] T dbdf08fc3dde4422aa60da02d73b3116 P 12d3554b22244952a8cb77fa76ad9e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:46.548385 5113 raft_consensus.cc:359] T dbdf08fc3dde4422aa60da02d73b3116 P ecdb77fe301541a4a0f652e78308dc80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:46.548385 5111 raft_consensus.cc:359] T dbdf08fc3dde4422aa60da02d73b3116 P f50d561a4167470b961a3f7dbeb6ef3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:46.548509 5111 raft_consensus.cc:385] T dbdf08fc3dde4422aa60da02d73b3116 P f50d561a4167470b961a3f7dbeb6ef3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:03:46.548509 5113 raft_consensus.cc:385] T dbdf08fc3dde4422aa60da02d73b3116 P ecdb77fe301541a4a0f652e78308dc80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:03:46.548550 5113 raft_consensus.cc:740] T dbdf08fc3dde4422aa60da02d73b3116 P ecdb77fe301541a4a0f652e78308dc80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecdb77fe301541a4a0f652e78308dc80, State: Initialized, Role: FOLLOWER
I20260429 14:03:46.548550 5111 raft_consensus.cc:740] T dbdf08fc3dde4422aa60da02d73b3116 P f50d561a4167470b961a3f7dbeb6ef3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f50d561a4167470b961a3f7dbeb6ef3e, State: Initialized, Role: FOLLOWER
I20260429 14:03:46.548632 5113 consensus_queue.cc:260] T dbdf08fc3dde4422aa60da02d73b3116 P ecdb77fe301541a4a0f652e78308dc80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:46.548632 5111 consensus_queue.cc:260] T dbdf08fc3dde4422aa60da02d73b3116 P f50d561a4167470b961a3f7dbeb6ef3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:46.548671 5112 ts_tablet_manager.cc:1434] T dbdf08fc3dde4422aa60da02d73b3116 P 12d3554b22244952a8cb77fa76ad9e38: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260429 14:03:46.548821 5088 heartbeater.cc:499] Master 127.2.197.254:36237 was elected leader, sending a full tablet report...
I20260429 14:03:46.548885 5111 ts_tablet_manager.cc:1434] T dbdf08fc3dde4422aa60da02d73b3116 P f50d561a4167470b961a3f7dbeb6ef3e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260429 14:03:46.548885 5113 ts_tablet_manager.cc:1434] T dbdf08fc3dde4422aa60da02d73b3116 P ecdb77fe301541a4a0f652e78308dc80: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260429 14:03:46.549034 4946 heartbeater.cc:499] Master 127.2.197.254:36237 was elected leader, sending a full tablet report...
I20260429 14:03:46.549229 4804 heartbeater.cc:499] Master 127.2.197.254:36237 was elected leader, sending a full tablet report...
W20260429 14:03:46.565207 4947 tablet.cc:2404] T dbdf08fc3dde4422aa60da02d73b3116 P f50d561a4167470b961a3f7dbeb6ef3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 14:03:46.576319 5118 raft_consensus.cc:493] T dbdf08fc3dde4422aa60da02d73b3116 P f50d561a4167470b961a3f7dbeb6ef3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:03:46.576485 5118 raft_consensus.cc:515] T dbdf08fc3dde4422aa60da02d73b3116 P f50d561a4167470b961a3f7dbeb6ef3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:46.576906 5118 leader_election.cc:290] T dbdf08fc3dde4422aa60da02d73b3116 P f50d561a4167470b961a3f7dbeb6ef3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecdb77fe301541a4a0f652e78308dc80 (127.2.197.193:42969), 12d3554b22244952a8cb77fa76ad9e38 (127.2.197.195:40127)
I20260429 14:03:46.579381 4839 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:46.577092 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:42969 (local address 127.2.197.194:59611)
0429 14:03:46.577182 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:46.577194 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:46.577242 (+ 48us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:46.577414 (+ 172us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:46.577416 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:46.577418 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:03:46.577614 (+ 196us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:46.577616 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.578482 (+ 866us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:46.578484 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:46.579090 (+ 606us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:46.579092 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.579195 (+ 103us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:46.579198 (+ 3us) client_negotiation.cc:770] Sending connection context
0429 14:03:46.579233 (+ 35us) client_negotiation.cc:241] Negotiation successful
0429 14:03:46.579274 (+ 41us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260429 14:03:46.579388 5120 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:46.577171 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:40127 (local address 127.2.197.194:52401)
0429 14:03:46.577334 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:46.577344 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:46.577415 (+ 71us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:46.577565 (+ 150us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:46.577566 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:46.577567 (+ 1us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:03:46.577786 (+ 219us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:46.577787 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.578501 (+ 714us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:46.578502 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:46.579085 (+ 583us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:46.579087 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.579192 (+ 105us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:46.579197 (+ 5us) client_negotiation.cc:770] Sending connection context
0429 14:03:46.579231 (+ 34us) client_negotiation.cc:241] Negotiation successful
0429 14:03:46.579270 (+ 39us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":116,"threads_started":1}
I20260429 14:03:46.579809 5108 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:46.577234 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:52401 (local address 127.2.197.195:40127)
0429 14:03:46.577285 (+ 51us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:46.577288 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:46.577411 (+ 123us) server_negotiation.cc:408] Connection header received
0429 14:03:46.577472 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:46.577473 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:46.577484 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:46.577559 (+ 75us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:03:46.577870 (+ 311us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.578416 (+ 546us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:46.579199 (+ 783us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.579712 (+ 513us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:46.579719 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:46.579739 (+ 20us) server_negotiation.cc:300] Negotiation successful
0429 14:03:46.579765 (+ 26us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260429 14:03:46.579833 5107 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:46.577187 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:59611 (local address 127.2.197.193:42969)
0429 14:03:46.577248 (+ 61us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:46.577251 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:46.577264 (+ 13us) server_negotiation.cc:408] Connection header received
0429 14:03:46.577297 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:46.577299 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:46.577308 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:46.577395 (+ 87us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:03:46.577703 (+ 308us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.578392 (+ 689us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:46.579201 (+ 809us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.579699 (+ 498us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:46.579708 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:46.579730 (+ 22us) server_negotiation.cc:300] Negotiation successful
0429 14:03:46.579761 (+ 31us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260429 14:03:46.580200 4758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdf08fc3dde4422aa60da02d73b3116" candidate_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecdb77fe301541a4a0f652e78308dc80" is_pre_election: true
I20260429 14:03:46.580200 5042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdf08fc3dde4422aa60da02d73b3116" candidate_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d3554b22244952a8cb77fa76ad9e38" is_pre_election: true
I20260429 14:03:46.580358 5042 raft_consensus.cc:2468] T dbdf08fc3dde4422aa60da02d73b3116 P 12d3554b22244952a8cb77fa76ad9e38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f50d561a4167470b961a3f7dbeb6ef3e in term 0.
I20260429 14:03:46.580358 4758 raft_consensus.cc:2468] T dbdf08fc3dde4422aa60da02d73b3116 P ecdb77fe301541a4a0f652e78308dc80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f50d561a4167470b961a3f7dbeb6ef3e in term 0.
I20260429 14:03:46.580633 4834 leader_election.cc:304] T dbdf08fc3dde4422aa60da02d73b3116 P f50d561a4167470b961a3f7dbeb6ef3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12d3554b22244952a8cb77fa76ad9e38, f50d561a4167470b961a3f7dbeb6ef3e; no voters:
I20260429 14:03:46.580853 5118 raft_consensus.cc:2804] T dbdf08fc3dde4422aa60da02d73b3116 P f50d561a4167470b961a3f7dbeb6ef3e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 14:03:46.580950 5118 raft_consensus.cc:493] T dbdf08fc3dde4422aa60da02d73b3116 P f50d561a4167470b961a3f7dbeb6ef3e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 14:03:46.580986 5118 raft_consensus.cc:3060] T dbdf08fc3dde4422aa60da02d73b3116 P f50d561a4167470b961a3f7dbeb6ef3e [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:03:46.581669 5118 raft_consensus.cc:515] T dbdf08fc3dde4422aa60da02d73b3116 P f50d561a4167470b961a3f7dbeb6ef3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:46.581861 5118 leader_election.cc:290] T dbdf08fc3dde4422aa60da02d73b3116 P f50d561a4167470b961a3f7dbeb6ef3e [CANDIDATE]: Term 1 election: Requested vote from peers ecdb77fe301541a4a0f652e78308dc80 (127.2.197.193:42969), 12d3554b22244952a8cb77fa76ad9e38 (127.2.197.195:40127)
I20260429 14:03:46.582046 5042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdf08fc3dde4422aa60da02d73b3116" candidate_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d3554b22244952a8cb77fa76ad9e38"
I20260429 14:03:46.582041 4758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbdf08fc3dde4422aa60da02d73b3116" candidate_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecdb77fe301541a4a0f652e78308dc80"
I20260429 14:03:46.582134 5042 raft_consensus.cc:3060] T dbdf08fc3dde4422aa60da02d73b3116 P 12d3554b22244952a8cb77fa76ad9e38 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:03:46.582131 4758 raft_consensus.cc:3060] T dbdf08fc3dde4422aa60da02d73b3116 P ecdb77fe301541a4a0f652e78308dc80 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:03:46.582974 4758 raft_consensus.cc:2468] T dbdf08fc3dde4422aa60da02d73b3116 P ecdb77fe301541a4a0f652e78308dc80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f50d561a4167470b961a3f7dbeb6ef3e in term 1.
I20260429 14:03:46.582998 5042 raft_consensus.cc:2468] T dbdf08fc3dde4422aa60da02d73b3116 P 12d3554b22244952a8cb77fa76ad9e38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f50d561a4167470b961a3f7dbeb6ef3e in term 1.
I20260429 14:03:46.583137 4834 leader_election.cc:304] T dbdf08fc3dde4422aa60da02d73b3116 P f50d561a4167470b961a3f7dbeb6ef3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12d3554b22244952a8cb77fa76ad9e38, f50d561a4167470b961a3f7dbeb6ef3e; no voters:
I20260429 14:03:46.583246 5118 raft_consensus.cc:2804] T dbdf08fc3dde4422aa60da02d73b3116 P f50d561a4167470b961a3f7dbeb6ef3e [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:03:46.583417 5118 raft_consensus.cc:697] T dbdf08fc3dde4422aa60da02d73b3116 P f50d561a4167470b961a3f7dbeb6ef3e [term 1 LEADER]: Becoming Leader. State: Replica: f50d561a4167470b961a3f7dbeb6ef3e, State: Running, Role: LEADER
I20260429 14:03:46.583537 5118 consensus_queue.cc:237] T dbdf08fc3dde4422aa60da02d73b3116 P f50d561a4167470b961a3f7dbeb6ef3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:46.584287 4599 catalog_manager.cc:5671] T dbdf08fc3dde4422aa60da02d73b3116 P f50d561a4167470b961a3f7dbeb6ef3e reported cstate change: term changed from 0 to 1, leader changed from <none> to f50d561a4167470b961a3f7dbeb6ef3e (127.2.197.194). New cstate: current_term: 1 leader_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } health_report { overall_health: UNKNOWN } } }
Apr 29 14:03:46 dist-test-slave-8n77 krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471425, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:03:46.622082 5128 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:46.615247 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:36237 (local address 127.0.0.1:51954)
0429 14:03:46.615401 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:46.615411 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:46.615464 (+ 53us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:46.615629 (+ 165us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:46.615630 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:46.615945 (+ 315us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:03:46.616106 (+ 161us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:46.616109 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.616869 (+ 760us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:46.616871 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:46.617254 (+ 383us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:46.617255 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.617318 (+ 63us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:46.617872 (+ 554us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:03:46.617877 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:03:46.619422 (+ 1545us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:03:46.621022 (+ 1600us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:46.621028 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:46.621031 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:46.621234 (+ 203us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:46.621418 (+ 184us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:46.621418 (+ 0us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:46.621419 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:46.621447 (+ 28us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:46.621733 (+ 286us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:03:46.621738 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:03:46.621785 (+ 47us) client_negotiation.cc:770] Sending connection context
0429 14:03:46.621851 (+ 66us) client_negotiation.cc:241] Negotiation successful
0429 14:03:46.622004 (+ 153us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
I20260429 14:03:46.622170 4648 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:46.615399 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51954 (local address 127.2.197.254:36237)
0429 14:03:46.615465 (+ 66us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:46.615467 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:46.615508 (+ 41us) server_negotiation.cc:408] Connection header received
0429 14:03:46.615532 (+ 24us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:46.615533 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:46.615544 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:46.615597 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:46.616240 (+ 643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.616756 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:46.617331 (+ 575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.617445 (+ 114us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:46.619516 (+ 2071us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:46.619521 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:46.619522 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:46.619528 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:46.620910 (+ 1382us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:46.621323 (+ 413us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:46.621324 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:46.621325 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:46.621348 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:46.621535 (+ 187us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:46.621536 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:46.621537 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:46.621637 (+ 100us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:46.621707 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:46.621993 (+ 286us) server_negotiation.cc:300] Negotiation successful
0429 14:03:46.622066 (+ 73us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260429 14:03:46.624264 4599 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:51954:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260429 14:03:46.624444 4599 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260429 14:03:46.625288 4805 tablet.cc:2404] T dbdf08fc3dde4422aa60da02d73b3116 P ecdb77fe301541a4a0f652e78308dc80: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 14:03:46.626372 4738 tablet_service.cc:1511] Processing CreateTablet for tablet beeb7aac685b4dc6bf9ba8bd2e757b5c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1553b57737aa43b39c1a40d1205f1942]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260429 14:03:46.626435 5022 tablet_service.cc:1511] Processing CreateTablet for tablet beeb7aac685b4dc6bf9ba8bd2e757b5c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1553b57737aa43b39c1a40d1205f1942]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260429 14:03:46.626544 4738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet beeb7aac685b4dc6bf9ba8bd2e757b5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:03:46.626582 5022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet beeb7aac685b4dc6bf9ba8bd2e757b5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:03:46.628158 5113 tablet_bootstrap.cc:492] T beeb7aac685b4dc6bf9ba8bd2e757b5c P ecdb77fe301541a4a0f652e78308dc80: Bootstrap starting.
I20260429 14:03:46.628294 4880 tablet_service.cc:1511] Processing CreateTablet for tablet beeb7aac685b4dc6bf9ba8bd2e757b5c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1553b57737aa43b39c1a40d1205f1942]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260429 14:03:46.628464 4880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet beeb7aac685b4dc6bf9ba8bd2e757b5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:03:46.628640 5113 tablet_bootstrap.cc:654] T beeb7aac685b4dc6bf9ba8bd2e757b5c P ecdb77fe301541a4a0f652e78308dc80: Neither blocks nor log segments found. Creating new log.
I20260429 14:03:46.628988 5112 tablet_bootstrap.cc:492] T beeb7aac685b4dc6bf9ba8bd2e757b5c P 12d3554b22244952a8cb77fa76ad9e38: Bootstrap starting.
I20260429 14:03:46.629351 5113 tablet_bootstrap.cc:492] T beeb7aac685b4dc6bf9ba8bd2e757b5c P ecdb77fe301541a4a0f652e78308dc80: No bootstrap required, opened a new log
I20260429 14:03:46.629408 5113 ts_tablet_manager.cc:1403] T beeb7aac685b4dc6bf9ba8bd2e757b5c P ecdb77fe301541a4a0f652e78308dc80: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260429 14:03:46.629390 5112 tablet_bootstrap.cc:654] T beeb7aac685b4dc6bf9ba8bd2e757b5c P 12d3554b22244952a8cb77fa76ad9e38: Neither blocks nor log segments found. Creating new log.
I20260429 14:03:46.629570 5113 raft_consensus.cc:359] T beeb7aac685b4dc6bf9ba8bd2e757b5c P ecdb77fe301541a4a0f652e78308dc80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:46.629671 5113 raft_consensus.cc:385] T beeb7aac685b4dc6bf9ba8bd2e757b5c P ecdb77fe301541a4a0f652e78308dc80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:03:46.629693 5113 raft_consensus.cc:740] T beeb7aac685b4dc6bf9ba8bd2e757b5c P ecdb77fe301541a4a0f652e78308dc80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecdb77fe301541a4a0f652e78308dc80, State: Initialized, Role: FOLLOWER
I20260429 14:03:46.629734 5113 consensus_queue.cc:260] T beeb7aac685b4dc6bf9ba8bd2e757b5c P ecdb77fe301541a4a0f652e78308dc80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:46.629827 5113 ts_tablet_manager.cc:1434] T beeb7aac685b4dc6bf9ba8bd2e757b5c P ecdb77fe301541a4a0f652e78308dc80: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260429 14:03:46.629853 5111 tablet_bootstrap.cc:492] T beeb7aac685b4dc6bf9ba8bd2e757b5c P f50d561a4167470b961a3f7dbeb6ef3e: Bootstrap starting.
I20260429 14:03:46.629949 5119 tablet_replica.cc:442] T beeb7aac685b4dc6bf9ba8bd2e757b5c P ecdb77fe301541a4a0f652e78308dc80: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } } }
I20260429 14:03:46.630010 5119 tablet_replica.cc:445] T beeb7aac685b4dc6bf9ba8bd2e757b5c P ecdb77fe301541a4a0f652e78308dc80: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 14:03:46.630189 5112 tablet_bootstrap.cc:492] T beeb7aac685b4dc6bf9ba8bd2e757b5c P 12d3554b22244952a8cb77fa76ad9e38: No bootstrap required, opened a new log
I20260429 14:03:46.630234 5112 ts_tablet_manager.cc:1403] T beeb7aac685b4dc6bf9ba8bd2e757b5c P 12d3554b22244952a8cb77fa76ad9e38: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260429 14:03:46.630304 5111 tablet_bootstrap.cc:654] T beeb7aac685b4dc6bf9ba8bd2e757b5c P f50d561a4167470b961a3f7dbeb6ef3e: Neither blocks nor log segments found. Creating new log.
I20260429 14:03:46.630429 5112 raft_consensus.cc:359] T beeb7aac685b4dc6bf9ba8bd2e757b5c P 12d3554b22244952a8cb77fa76ad9e38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:46.630496 5112 raft_consensus.cc:385] T beeb7aac685b4dc6bf9ba8bd2e757b5c P 12d3554b22244952a8cb77fa76ad9e38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:03:46.630522 5112 raft_consensus.cc:740] T beeb7aac685b4dc6bf9ba8bd2e757b5c P 12d3554b22244952a8cb77fa76ad9e38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12d3554b22244952a8cb77fa76ad9e38, State: Initialized, Role: FOLLOWER
I20260429 14:03:46.630591 5112 consensus_queue.cc:260] T beeb7aac685b4dc6bf9ba8bd2e757b5c P 12d3554b22244952a8cb77fa76ad9e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:46.630692 5112 ts_tablet_manager.cc:1434] T beeb7aac685b4dc6bf9ba8bd2e757b5c P 12d3554b22244952a8cb77fa76ad9e38: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:46.630751 5117 tablet_replica.cc:442] T beeb7aac685b4dc6bf9ba8bd2e757b5c P 12d3554b22244952a8cb77fa76ad9e38: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } } }
I20260429 14:03:46.630865 5117 tablet_replica.cc:445] T beeb7aac685b4dc6bf9ba8bd2e757b5c P 12d3554b22244952a8cb77fa76ad9e38: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 14:03:46.631392 5111 tablet_bootstrap.cc:492] T beeb7aac685b4dc6bf9ba8bd2e757b5c P f50d561a4167470b961a3f7dbeb6ef3e: No bootstrap required, opened a new log
I20260429 14:03:46.631445 5111 ts_tablet_manager.cc:1403] T beeb7aac685b4dc6bf9ba8bd2e757b5c P f50d561a4167470b961a3f7dbeb6ef3e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260429 14:03:46.631605 5111 raft_consensus.cc:359] T beeb7aac685b4dc6bf9ba8bd2e757b5c P f50d561a4167470b961a3f7dbeb6ef3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:46.631680 5111 raft_consensus.cc:385] T beeb7aac685b4dc6bf9ba8bd2e757b5c P f50d561a4167470b961a3f7dbeb6ef3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:03:46.631701 5111 raft_consensus.cc:740] T beeb7aac685b4dc6bf9ba8bd2e757b5c P f50d561a4167470b961a3f7dbeb6ef3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f50d561a4167470b961a3f7dbeb6ef3e, State: Initialized, Role: FOLLOWER
I20260429 14:03:46.631743 5111 consensus_queue.cc:260] T beeb7aac685b4dc6bf9ba8bd2e757b5c P f50d561a4167470b961a3f7dbeb6ef3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:46.631839 5111 ts_tablet_manager.cc:1434] T beeb7aac685b4dc6bf9ba8bd2e757b5c P f50d561a4167470b961a3f7dbeb6ef3e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:46.631887 5118 tablet_replica.cc:442] T beeb7aac685b4dc6bf9ba8bd2e757b5c P f50d561a4167470b961a3f7dbeb6ef3e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } } }
I20260429 14:03:46.631990 5118 tablet_replica.cc:445] T beeb7aac685b4dc6bf9ba8bd2e757b5c P f50d561a4167470b961a3f7dbeb6ef3e: This TxnStatusTablet replica's current role is: FOLLOWER
W20260429 14:03:46.747959 5089 tablet.cc:2404] T beeb7aac685b4dc6bf9ba8bd2e757b5c P 12d3554b22244952a8cb77fa76ad9e38: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 14:03:46.773276 5117 raft_consensus.cc:493] T beeb7aac685b4dc6bf9ba8bd2e757b5c P 12d3554b22244952a8cb77fa76ad9e38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:03:46.773447 5117 raft_consensus.cc:515] T beeb7aac685b4dc6bf9ba8bd2e757b5c P 12d3554b22244952a8cb77fa76ad9e38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:46.773957 5117 leader_election.cc:290] T beeb7aac685b4dc6bf9ba8bd2e757b5c P 12d3554b22244952a8cb77fa76ad9e38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecdb77fe301541a4a0f652e78308dc80 (127.2.197.193:42969), f50d561a4167470b961a3f7dbeb6ef3e (127.2.197.194:37203)
I20260429 14:03:46.777040 4981 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:46.774137 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:42969 (local address 127.2.197.195:47573)
0429 14:03:46.774205 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:46.774218 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:46.774264 (+ 46us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:46.774468 (+ 204us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:46.774469 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:46.774471 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:03:46.774741 (+ 270us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:46.774744 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.775586 (+ 842us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:46.775587 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:46.776653 (+ 1066us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:46.776659 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.776820 (+ 161us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:46.776827 (+ 7us) client_negotiation.cc:770] Sending connection context
0429 14:03:46.776903 (+ 76us) client_negotiation.cc:241] Negotiation successful
0429 14:03:46.776961 (+ 58us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260429 14:03:46.777051 5132 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:46.774296 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:37203 (local address 127.2.197.195:54365)
0429 14:03:46.774508 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:46.774519 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:46.774588 (+ 69us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:46.774823 (+ 235us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:46.774824 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:46.774826 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:03:46.775091 (+ 265us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:46.775093 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.776115 (+ 1022us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:46.776116 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:46.776751 (+ 635us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:46.776753 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.776876 (+ 123us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:46.776880 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 14:03:46.776906 (+ 26us) client_negotiation.cc:241] Negotiation successful
0429 14:03:46.776947 (+ 41us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":102,"threads_started":1}
I20260429 14:03:46.777463 5107 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:46.774252 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:47573 (local address 127.2.197.193:42969)
0429 14:03:46.774323 (+ 71us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:46.774326 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:46.774336 (+ 10us) server_negotiation.cc:408] Connection header received
0429 14:03:46.774366 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:46.774367 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:46.774377 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:46.774440 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:03:46.774853 (+ 413us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.775471 (+ 618us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:46.776854 (+ 1383us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.777353 (+ 499us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:46.777362 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:46.777393 (+ 31us) server_negotiation.cc:300] Negotiation successful
0429 14:03:46.777419 (+ 26us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260429 14:03:46.777583 5110 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:46.774389 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:54365 (local address 127.2.197.194:37203)
0429 14:03:46.774440 (+ 51us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:46.774442 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:46.774625 (+ 183us) server_negotiation.cc:408] Connection header received
0429 14:03:46.774652 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:46.774653 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:46.774662 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:46.774718 (+ 56us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:03:46.775212 (+ 494us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.775983 (+ 771us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:46.776921 (+ 938us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.777447 (+ 526us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:46.777455 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:46.777478 (+ 23us) server_negotiation.cc:300] Negotiation successful
0429 14:03:46.777513 (+ 35us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260429 14:03:46.777724 4758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "beeb7aac685b4dc6bf9ba8bd2e757b5c" candidate_uuid: "12d3554b22244952a8cb77fa76ad9e38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecdb77fe301541a4a0f652e78308dc80" is_pre_election: true
I20260429 14:03:46.777824 4758 raft_consensus.cc:2468] T beeb7aac685b4dc6bf9ba8bd2e757b5c P ecdb77fe301541a4a0f652e78308dc80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12d3554b22244952a8cb77fa76ad9e38 in term 0.
I20260429 14:03:46.777944 4900 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "beeb7aac685b4dc6bf9ba8bd2e757b5c" candidate_uuid: "12d3554b22244952a8cb77fa76ad9e38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" is_pre_election: true
I20260429 14:03:46.778076 4900 raft_consensus.cc:2468] T beeb7aac685b4dc6bf9ba8bd2e757b5c P f50d561a4167470b961a3f7dbeb6ef3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12d3554b22244952a8cb77fa76ad9e38 in term 0.
I20260429 14:03:46.778060 4974 leader_election.cc:304] T beeb7aac685b4dc6bf9ba8bd2e757b5c P 12d3554b22244952a8cb77fa76ad9e38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12d3554b22244952a8cb77fa76ad9e38, ecdb77fe301541a4a0f652e78308dc80; no voters:
I20260429 14:03:46.778220 5117 raft_consensus.cc:2804] T beeb7aac685b4dc6bf9ba8bd2e757b5c P 12d3554b22244952a8cb77fa76ad9e38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 14:03:46.778324 5117 raft_consensus.cc:493] T beeb7aac685b4dc6bf9ba8bd2e757b5c P 12d3554b22244952a8cb77fa76ad9e38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 14:03:46.778362 5117 raft_consensus.cc:3060] T beeb7aac685b4dc6bf9ba8bd2e757b5c P 12d3554b22244952a8cb77fa76ad9e38 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:03:46.779500 5117 raft_consensus.cc:515] T beeb7aac685b4dc6bf9ba8bd2e757b5c P 12d3554b22244952a8cb77fa76ad9e38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:46.779911 5117 leader_election.cc:290] T beeb7aac685b4dc6bf9ba8bd2e757b5c P 12d3554b22244952a8cb77fa76ad9e38 [CANDIDATE]: Term 1 election: Requested vote from peers ecdb77fe301541a4a0f652e78308dc80 (127.2.197.193:42969), f50d561a4167470b961a3f7dbeb6ef3e (127.2.197.194:37203)
I20260429 14:03:46.780318 4900 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "beeb7aac685b4dc6bf9ba8bd2e757b5c" candidate_uuid: "12d3554b22244952a8cb77fa76ad9e38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f50d561a4167470b961a3f7dbeb6ef3e"
I20260429 14:03:46.780316 4758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "beeb7aac685b4dc6bf9ba8bd2e757b5c" candidate_uuid: "12d3554b22244952a8cb77fa76ad9e38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecdb77fe301541a4a0f652e78308dc80"
I20260429 14:03:46.780432 4758 raft_consensus.cc:3060] T beeb7aac685b4dc6bf9ba8bd2e757b5c P ecdb77fe301541a4a0f652e78308dc80 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:03:46.780435 4900 raft_consensus.cc:3060] T beeb7aac685b4dc6bf9ba8bd2e757b5c P f50d561a4167470b961a3f7dbeb6ef3e [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:03:46.781103 4900 raft_consensus.cc:2468] T beeb7aac685b4dc6bf9ba8bd2e757b5c P f50d561a4167470b961a3f7dbeb6ef3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12d3554b22244952a8cb77fa76ad9e38 in term 1.
I20260429 14:03:46.781111 4758 raft_consensus.cc:2468] T beeb7aac685b4dc6bf9ba8bd2e757b5c P ecdb77fe301541a4a0f652e78308dc80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12d3554b22244952a8cb77fa76ad9e38 in term 1.
I20260429 14:03:46.781332 4974 leader_election.cc:304] T beeb7aac685b4dc6bf9ba8bd2e757b5c P 12d3554b22244952a8cb77fa76ad9e38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12d3554b22244952a8cb77fa76ad9e38, f50d561a4167470b961a3f7dbeb6ef3e; no voters:
I20260429 14:03:46.781435 5117 raft_consensus.cc:2804] T beeb7aac685b4dc6bf9ba8bd2e757b5c P 12d3554b22244952a8cb77fa76ad9e38 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:03:46.781656 5117 raft_consensus.cc:697] T beeb7aac685b4dc6bf9ba8bd2e757b5c P 12d3554b22244952a8cb77fa76ad9e38 [term 1 LEADER]: Becoming Leader. State: Replica: 12d3554b22244952a8cb77fa76ad9e38, State: Running, Role: LEADER
I20260429 14:03:46.781760 5117 consensus_queue.cc:237] T beeb7aac685b4dc6bf9ba8bd2e757b5c P 12d3554b22244952a8cb77fa76ad9e38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:46.782373 5133 tablet_replica.cc:442] T beeb7aac685b4dc6bf9ba8bd2e757b5c P 12d3554b22244952a8cb77fa76ad9e38: TxnStatusTablet state changed. Reason: New leader 12d3554b22244952a8cb77fa76ad9e38. Latest consensus state: current_term: 1 leader_uuid: "12d3554b22244952a8cb77fa76ad9e38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } } }
I20260429 14:03:46.782464 5133 tablet_replica.cc:445] T beeb7aac685b4dc6bf9ba8bd2e757b5c P 12d3554b22244952a8cb77fa76ad9e38: This TxnStatusTablet replica's current role is: LEADER
I20260429 14:03:46.782660 5135 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260429 14:03:46.782788 5135 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260429 14:03:46.782869 4599 catalog_manager.cc:5671] T beeb7aac685b4dc6bf9ba8bd2e757b5c P 12d3554b22244952a8cb77fa76ad9e38 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12d3554b22244952a8cb77fa76ad9e38 (127.2.197.195). New cstate: current_term: 1 leader_uuid: "12d3554b22244952a8cb77fa76ad9e38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } health_report { overall_health: HEALTHY } } }
I20260429 14:03:46.814769 5108 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:46.812436 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49442 (local address 127.2.197.195:40127)
0429 14:03:46.812507 (+ 71us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:46.812509 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:46.812555 (+ 46us) server_negotiation.cc:408] Connection header received
0429 14:03:46.812595 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:46.812597 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:46.812621 (+ 24us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:46.812690 (+ 69us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:03:46.813002 (+ 312us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.813464 (+ 462us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:46.814120 (+ 656us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.814301 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:46.814352 (+ 51us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:03:46.814459 (+ 107us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:03:46.814557 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:46.814657 (+ 100us) server_negotiation.cc:300] Negotiation successful
0429 14:03:46.814686 (+ 29us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260429 14:03:46.814783 5128 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:46.812348 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:40127 (local address 127.0.0.1:49442)
0429 14:03:46.812418 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:46.812429 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:46.812494 (+ 65us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:46.812700 (+ 206us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:46.812701 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:46.812702 (+ 1us) client_negotiation.cc:190] Negotiated authn=TOKEN
0429 14:03:46.812873 (+ 171us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:46.812875 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.813565 (+ 690us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:46.813566 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:46.814026 (+ 460us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:46.814028 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.814121 (+ 93us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:46.814139 (+ 18us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0429 14:03:46.814551 (+ 412us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0429 14:03:46.814559 (+ 8us) client_negotiation.cc:770] Sending connection context
0429 14:03:46.814653 (+ 94us) client_negotiation.cc:241] Negotiation successful
0429 14:03:46.814709 (+ 56us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260429 14:03:46.816879 4900 raft_consensus.cc:1275] T beeb7aac685b4dc6bf9ba8bd2e757b5c P f50d561a4167470b961a3f7dbeb6ef3e [term 1 FOLLOWER]: Refusing update from remote peer 12d3554b22244952a8cb77fa76ad9e38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:03:46.816895 4758 raft_consensus.cc:1275] T beeb7aac685b4dc6bf9ba8bd2e757b5c P ecdb77fe301541a4a0f652e78308dc80 [term 1 FOLLOWER]: Refusing update from remote peer 12d3554b22244952a8cb77fa76ad9e38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:03:46.817224 5133 consensus_queue.cc:1048] T beeb7aac685b4dc6bf9ba8bd2e757b5c P 12d3554b22244952a8cb77fa76ad9e38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:03:46.817382 5117 consensus_queue.cc:1048] T beeb7aac685b4dc6bf9ba8bd2e757b5c P 12d3554b22244952a8cb77fa76ad9e38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:03:46.818435 5119 tablet_replica.cc:442] T beeb7aac685b4dc6bf9ba8bd2e757b5c P ecdb77fe301541a4a0f652e78308dc80: TxnStatusTablet state changed. Reason: New leader 12d3554b22244952a8cb77fa76ad9e38. Latest consensus state: current_term: 1 leader_uuid: "12d3554b22244952a8cb77fa76ad9e38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } } }
I20260429 14:03:46.818435 5118 tablet_replica.cc:442] T beeb7aac685b4dc6bf9ba8bd2e757b5c P f50d561a4167470b961a3f7dbeb6ef3e: TxnStatusTablet state changed. Reason: New leader 12d3554b22244952a8cb77fa76ad9e38. Latest consensus state: current_term: 1 leader_uuid: "12d3554b22244952a8cb77fa76ad9e38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } } }
I20260429 14:03:46.818518 5118 tablet_replica.cc:445] T beeb7aac685b4dc6bf9ba8bd2e757b5c P f50d561a4167470b961a3f7dbeb6ef3e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 14:03:46.818518 5119 tablet_replica.cc:445] T beeb7aac685b4dc6bf9ba8bd2e757b5c P ecdb77fe301541a4a0f652e78308dc80: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 14:03:46.819665 5133 tablet_replica.cc:442] T beeb7aac685b4dc6bf9ba8bd2e757b5c P 12d3554b22244952a8cb77fa76ad9e38: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "12d3554b22244952a8cb77fa76ad9e38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } } }
I20260429 14:03:46.819756 5133 tablet_replica.cc:445] T beeb7aac685b4dc6bf9ba8bd2e757b5c P 12d3554b22244952a8cb77fa76ad9e38: This TxnStatusTablet replica's current role is: LEADER
I20260429 14:03:46.819737 5117 tablet_replica.cc:442] T beeb7aac685b4dc6bf9ba8bd2e757b5c P 12d3554b22244952a8cb77fa76ad9e38: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "12d3554b22244952a8cb77fa76ad9e38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } } }
I20260429 14:03:46.819799 5117 tablet_replica.cc:445] T beeb7aac685b4dc6bf9ba8bd2e757b5c P 12d3554b22244952a8cb77fa76ad9e38: This TxnStatusTablet replica's current role is: LEADER
I20260429 14:03:46.819761 5119 tablet_replica.cc:442] T beeb7aac685b4dc6bf9ba8bd2e757b5c P ecdb77fe301541a4a0f652e78308dc80: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "12d3554b22244952a8cb77fa76ad9e38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } } }
I20260429 14:03:46.819844 5119 tablet_replica.cc:445] T beeb7aac685b4dc6bf9ba8bd2e757b5c P ecdb77fe301541a4a0f652e78308dc80: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 14:03:46.820006 5140 mvcc.cc:204] Tried to move back new op lower bound from 7280522964238950400 to 7280522964099223552. Current Snapshot: MvccSnapshot[applied={T|T < 7280522964238950400}]
I20260429 14:03:46.820087 5138 mvcc.cc:204] Tried to move back new op lower bound from 7280522964238950400 to 7280522964099223552. Current Snapshot: MvccSnapshot[applied={T|T < 7280522964238950400}]
I20260429 14:03:46.820611 5118 tablet_replica.cc:442] T beeb7aac685b4dc6bf9ba8bd2e757b5c P f50d561a4167470b961a3f7dbeb6ef3e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "12d3554b22244952a8cb77fa76ad9e38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } } }
I20260429 14:03:46.820703 5118 tablet_replica.cc:445] T beeb7aac685b4dc6bf9ba8bd2e757b5c P f50d561a4167470b961a3f7dbeb6ef3e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 14:03:46.820819 5139 mvcc.cc:204] Tried to move back new op lower bound from 7280522964238950400 to 7280522964099223552. Current Snapshot: MvccSnapshot[applied={T|T < 7280522964238950400}]
I20260429 14:03:46.825870 5110 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:46.823355 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:45914 (local address 127.2.197.194:37203)
0429 14:03:46.823393 (+ 38us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:46.823395 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:46.823404 (+ 9us) server_negotiation.cc:408] Connection header received
0429 14:03:46.823562 (+ 158us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:46.823563 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:46.823577 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:46.823642 (+ 65us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:03:46.824060 (+ 418us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.824584 (+ 524us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:46.825177 (+ 593us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.825325 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:46.825353 (+ 28us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:03:46.825433 (+ 80us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:03:46.825513 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:46.825585 (+ 72us) server_negotiation.cc:300] Negotiation successful
0429 14:03:46.825611 (+ 26us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260429 14:03:46.830971 5108 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:46.828556 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:48449 (local address 127.2.197.195:40127)
0429 14:03:46.828663 (+ 107us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:46.828665 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:46.828753 (+ 88us) server_negotiation.cc:408] Connection header received
0429 14:03:46.828826 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:46.828828 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:46.828837 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:46.828887 (+ 50us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:03:46.829255 (+ 368us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.829696 (+ 441us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:46.830561 (+ 865us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.830713 (+ 152us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:46.830731 (+ 18us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:03:46.830752 (+ 21us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:03:46.830804 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:46.830892 (+ 88us) server_negotiation.cc:300] Negotiation successful
0429 14:03:46.830913 (+ 21us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260429 14:03:46.831019 5150 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:46.828380 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:40127 (local address 127.2.197.194:48449)
0429 14:03:46.828671 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:46.828684 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:46.828756 (+ 72us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:46.828907 (+ 151us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:46.828908 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:46.828910 (+ 2us) client_negotiation.cc:190] Negotiated authn=TOKEN
0429 14:03:46.829154 (+ 244us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:46.829156 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.829784 (+ 628us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:46.829785 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:46.830469 (+ 684us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:46.830472 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.830548 (+ 76us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:46.830561 (+ 13us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0429 14:03:46.830827 (+ 266us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0429 14:03:46.830829 (+ 2us) client_negotiation.cc:770] Sending connection context
0429 14:03:46.830883 (+ 54us) client_negotiation.cc:241] Negotiation successful
0429 14:03:46.830928 (+ 45us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":216,"threads_started":1}
I20260429 14:03:46.833861 4758 raft_consensus.cc:1275] T dbdf08fc3dde4422aa60da02d73b3116 P ecdb77fe301541a4a0f652e78308dc80 [term 1 FOLLOWER]: Refusing update from remote peer f50d561a4167470b961a3f7dbeb6ef3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:03:46.833978 5042 raft_consensus.cc:1275] T dbdf08fc3dde4422aa60da02d73b3116 P 12d3554b22244952a8cb77fa76ad9e38 [term 1 FOLLOWER]: Refusing update from remote peer f50d561a4167470b961a3f7dbeb6ef3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:03:46.834142 5118 consensus_queue.cc:1048] T dbdf08fc3dde4422aa60da02d73b3116 P f50d561a4167470b961a3f7dbeb6ef3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:03:46.834424 5121 consensus_queue.cc:1048] T dbdf08fc3dde4422aa60da02d73b3116 P f50d561a4167470b961a3f7dbeb6ef3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:03:46.844022 5110 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:46.841593 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:37925 (local address 127.2.197.194:37203)
0429 14:03:46.841632 (+ 39us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:46.841634 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:46.841806 (+ 172us) server_negotiation.cc:408] Connection header received
0429 14:03:46.841834 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:46.841835 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:46.841846 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:46.841897 (+ 51us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:03:46.842395 (+ 498us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.842862 (+ 467us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:46.843660 (+ 798us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.843773 (+ 113us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:46.843789 (+ 16us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:03:46.843808 (+ 19us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:03:46.843859 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:46.843953 (+ 94us) server_negotiation.cc:300] Negotiation successful
0429 14:03:46.843969 (+ 16us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260429 14:03:46.844075 5094 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:46.841502 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:37203 (local address 127.2.197.195:37925)
0429 14:03:46.841613 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:46.841627 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:46.841685 (+ 58us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:46.842066 (+ 381us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:46.842067 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:46.842068 (+ 1us) client_negotiation.cc:190] Negotiated authn=TOKEN
0429 14:03:46.842309 (+ 241us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:46.842312 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.842941 (+ 629us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:46.842942 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:46.843573 (+ 631us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:46.843575 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:46.843666 (+ 91us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:46.843680 (+ 14us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0429 14:03:46.843886 (+ 206us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0429 14:03:46.843890 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 14:03:46.843954 (+ 64us) client_negotiation.cc:241] Negotiation successful
0429 14:03:46.843998 (+ 44us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260429 14:03:46.855259 4597 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:03:46.855396 4597 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:03:46.856236 4597 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 1aeac3f977874eb79cf735db9587a3eb: Sending DeleteTablet for 3 replicas of tablet dbdf08fc3dde4422aa60da02d73b3116
I20260429 14:03:46.856644 4738 tablet_service.cc:1558] Processing DeleteTablet for tablet dbdf08fc3dde4422aa60da02d73b3116 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:03:46 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:52900
I20260429 14:03:46.856920 5157 tablet_replica.cc:333] T dbdf08fc3dde4422aa60da02d73b3116 P ecdb77fe301541a4a0f652e78308dc80: stopping tablet replica
I20260429 14:03:46.856971 4880 tablet_service.cc:1558] Processing DeleteTablet for tablet dbdf08fc3dde4422aa60da02d73b3116 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:03:46 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:45904
I20260429 14:03:46.857144 5157 raft_consensus.cc:2243] T dbdf08fc3dde4422aa60da02d73b3116 P ecdb77fe301541a4a0f652e78308dc80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:03:46.857255 5157 raft_consensus.cc:2272] T dbdf08fc3dde4422aa60da02d73b3116 P ecdb77fe301541a4a0f652e78308dc80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:03:46.857237 5022 tablet_service.cc:1558] Processing DeleteTablet for tablet dbdf08fc3dde4422aa60da02d73b3116 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:03:46 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:49430
I20260429 14:03:46.857453 4597 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 14:03:46.857597 4597 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 14:03:46.858026 5159 tablet_replica.cc:333] T dbdf08fc3dde4422aa60da02d73b3116 P 12d3554b22244952a8cb77fa76ad9e38: stopping tablet replica
I20260429 14:03:46.858107 5159 raft_consensus.cc:2243] T dbdf08fc3dde4422aa60da02d73b3116 P 12d3554b22244952a8cb77fa76ad9e38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:03:46.858165 5159 raft_consensus.cc:2272] T dbdf08fc3dde4422aa60da02d73b3116 P 12d3554b22244952a8cb77fa76ad9e38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:03:46.857815 5158 tablet_replica.cc:333] T dbdf08fc3dde4422aa60da02d73b3116 P f50d561a4167470b961a3f7dbeb6ef3e: stopping tablet replica
I20260429 14:03:46.858322 5158 raft_consensus.cc:2243] T dbdf08fc3dde4422aa60da02d73b3116 P f50d561a4167470b961a3f7dbeb6ef3e [term 1 LEADER]: Raft consensus shutting down.
I20260429 14:03:46.858422 5158 raft_consensus.cc:2272] T dbdf08fc3dde4422aa60da02d73b3116 P f50d561a4167470b961a3f7dbeb6ef3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:03:46.858903 5159 ts_tablet_manager.cc:1916] T dbdf08fc3dde4422aa60da02d73b3116 P 12d3554b22244952a8cb77fa76ad9e38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:03:46.859656 4738 tablet_service.cc:1511] Processing CreateTablet for tablet d1f7e5e193714b738cc629970ae5c054 (DEFAULT_TABLE table=test-table [id=66ec6fa1d6164ff3a92943d60d93e28c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:03:46.859781 4738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1f7e5e193714b738cc629970ae5c054. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:03:46.860953 5158 ts_tablet_manager.cc:1916] T dbdf08fc3dde4422aa60da02d73b3116 P f50d561a4167470b961a3f7dbeb6ef3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:03:46.861089 5113 tablet_bootstrap.cc:492] T d1f7e5e193714b738cc629970ae5c054 P ecdb77fe301541a4a0f652e78308dc80: Bootstrap starting.
I20260429 14:03:46.861147 4880 tablet_service.cc:1511] Processing CreateTablet for tablet d1f7e5e193714b738cc629970ae5c054 (DEFAULT_TABLE table=test-table [id=66ec6fa1d6164ff3a92943d60d93e28c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:03:46.861251 4880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1f7e5e193714b738cc629970ae5c054. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:03:46.861737 5113 tablet_bootstrap.cc:654] T d1f7e5e193714b738cc629970ae5c054 P ecdb77fe301541a4a0f652e78308dc80: Neither blocks nor log segments found. Creating new log.
I20260429 14:03:46.862594 5022 tablet_service.cc:1511] Processing CreateTablet for tablet d1f7e5e193714b738cc629970ae5c054 (DEFAULT_TABLE table=test-table [id=66ec6fa1d6164ff3a92943d60d93e28c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:03:46.862684 5158 ts_tablet_manager.cc:1929] T dbdf08fc3dde4422aa60da02d73b3116 P f50d561a4167470b961a3f7dbeb6ef3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 14:03:46.862727 5022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1f7e5e193714b738cc629970ae5c054. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:03:46.862749 5158 log.cc:1199] T dbdf08fc3dde4422aa60da02d73b3116 P f50d561a4167470b961a3f7dbeb6ef3e: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-1/wal/wals/dbdf08fc3dde4422aa60da02d73b3116
I20260429 14:03:46.863026 5158 ts_tablet_manager.cc:1950] T dbdf08fc3dde4422aa60da02d73b3116 P f50d561a4167470b961a3f7dbeb6ef3e: Deleting consensus metadata
I20260429 14:03:46.862618 5159 ts_tablet_manager.cc:1929] T dbdf08fc3dde4422aa60da02d73b3116 P 12d3554b22244952a8cb77fa76ad9e38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 14:03:46.863198 5159 log.cc:1199] T dbdf08fc3dde4422aa60da02d73b3116 P 12d3554b22244952a8cb77fa76ad9e38: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-2/wal/wals/dbdf08fc3dde4422aa60da02d73b3116
I20260429 14:03:46.863451 5159 ts_tablet_manager.cc:1950] T dbdf08fc3dde4422aa60da02d73b3116 P 12d3554b22244952a8cb77fa76ad9e38: Deleting consensus metadata
I20260429 14:03:46.863878 4586 catalog_manager.cc:5002] TS 12d3554b22244952a8cb77fa76ad9e38 (127.2.197.195:40127): tablet dbdf08fc3dde4422aa60da02d73b3116 (table test-table [id=4d0f06ca6b0e48878392699d4106439a]) successfully deleted
I20260429 14:03:46.863945 5112 tablet_bootstrap.cc:492] T d1f7e5e193714b738cc629970ae5c054 P 12d3554b22244952a8cb77fa76ad9e38: Bootstrap starting.
I20260429 14:03:46.863977 5111 tablet_bootstrap.cc:492] T d1f7e5e193714b738cc629970ae5c054 P f50d561a4167470b961a3f7dbeb6ef3e: Bootstrap starting.
I20260429 14:03:46.864346 5112 tablet_bootstrap.cc:654] T d1f7e5e193714b738cc629970ae5c054 P 12d3554b22244952a8cb77fa76ad9e38: Neither blocks nor log segments found. Creating new log.
I20260429 14:03:46.864430 5111 tablet_bootstrap.cc:654] T d1f7e5e193714b738cc629970ae5c054 P f50d561a4167470b961a3f7dbeb6ef3e: Neither blocks nor log segments found. Creating new log.
I20260429 14:03:46.864998 5112 tablet_bootstrap.cc:492] T d1f7e5e193714b738cc629970ae5c054 P 12d3554b22244952a8cb77fa76ad9e38: No bootstrap required, opened a new log
I20260429 14:03:46.865062 5112 ts_tablet_manager.cc:1403] T d1f7e5e193714b738cc629970ae5c054 P 12d3554b22244952a8cb77fa76ad9e38: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260429 14:03:46.865065 4584 catalog_manager.cc:5002] TS f50d561a4167470b961a3f7dbeb6ef3e (127.2.197.194:37203): tablet dbdf08fc3dde4422aa60da02d73b3116 (table test-table [id=4d0f06ca6b0e48878392699d4106439a]) successfully deleted
I20260429 14:03:46.865206 5112 raft_consensus.cc:359] T d1f7e5e193714b738cc629970ae5c054 P 12d3554b22244952a8cb77fa76ad9e38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:46.865267 5112 raft_consensus.cc:385] T d1f7e5e193714b738cc629970ae5c054 P 12d3554b22244952a8cb77fa76ad9e38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:03:46.865288 5112 raft_consensus.cc:740] T d1f7e5e193714b738cc629970ae5c054 P 12d3554b22244952a8cb77fa76ad9e38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12d3554b22244952a8cb77fa76ad9e38, State: Initialized, Role: FOLLOWER
I20260429 14:03:46.865330 5112 consensus_queue.cc:260] T d1f7e5e193714b738cc629970ae5c054 P 12d3554b22244952a8cb77fa76ad9e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:46.865412 5112 ts_tablet_manager.cc:1434] T d1f7e5e193714b738cc629970ae5c054 P 12d3554b22244952a8cb77fa76ad9e38: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:46.866556 5157 ts_tablet_manager.cc:1916] T dbdf08fc3dde4422aa60da02d73b3116 P ecdb77fe301541a4a0f652e78308dc80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:03:46.868854 5157 ts_tablet_manager.cc:1929] T dbdf08fc3dde4422aa60da02d73b3116 P ecdb77fe301541a4a0f652e78308dc80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 14:03:46.869001 5157 log.cc:1199] T dbdf08fc3dde4422aa60da02d73b3116 P ecdb77fe301541a4a0f652e78308dc80: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-0/wal/wals/dbdf08fc3dde4422aa60da02d73b3116
I20260429 14:03:46.869356 5157 ts_tablet_manager.cc:1950] T dbdf08fc3dde4422aa60da02d73b3116 P ecdb77fe301541a4a0f652e78308dc80: Deleting consensus metadata
I20260429 14:03:46.869831 4584 catalog_manager.cc:5002] TS ecdb77fe301541a4a0f652e78308dc80 (127.2.197.193:42969): tablet dbdf08fc3dde4422aa60da02d73b3116 (table test-table [id=4d0f06ca6b0e48878392699d4106439a]) successfully deleted
I20260429 14:03:46.871305 5111 tablet_bootstrap.cc:492] T d1f7e5e193714b738cc629970ae5c054 P f50d561a4167470b961a3f7dbeb6ef3e: No bootstrap required, opened a new log
I20260429 14:03:46.871368 5111 ts_tablet_manager.cc:1403] T d1f7e5e193714b738cc629970ae5c054 P f50d561a4167470b961a3f7dbeb6ef3e: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260429 14:03:46.871546 5111 raft_consensus.cc:359] T d1f7e5e193714b738cc629970ae5c054 P f50d561a4167470b961a3f7dbeb6ef3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:46.871624 5111 raft_consensus.cc:385] T d1f7e5e193714b738cc629970ae5c054 P f50d561a4167470b961a3f7dbeb6ef3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:03:46.871646 5111 raft_consensus.cc:740] T d1f7e5e193714b738cc629970ae5c054 P f50d561a4167470b961a3f7dbeb6ef3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f50d561a4167470b961a3f7dbeb6ef3e, State: Initialized, Role: FOLLOWER
I20260429 14:03:46.871691 5111 consensus_queue.cc:260] T d1f7e5e193714b738cc629970ae5c054 P f50d561a4167470b961a3f7dbeb6ef3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:46.871775 5111 ts_tablet_manager.cc:1434] T d1f7e5e193714b738cc629970ae5c054 P f50d561a4167470b961a3f7dbeb6ef3e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:46.874778 5113 tablet_bootstrap.cc:492] T d1f7e5e193714b738cc629970ae5c054 P ecdb77fe301541a4a0f652e78308dc80: No bootstrap required, opened a new log
I20260429 14:03:46.875054 5113 ts_tablet_manager.cc:1403] T d1f7e5e193714b738cc629970ae5c054 P ecdb77fe301541a4a0f652e78308dc80: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.000s
I20260429 14:03:46.875315 5113 raft_consensus.cc:359] T d1f7e5e193714b738cc629970ae5c054 P ecdb77fe301541a4a0f652e78308dc80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:46.875641 5113 raft_consensus.cc:385] T d1f7e5e193714b738cc629970ae5c054 P ecdb77fe301541a4a0f652e78308dc80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:03:46.875710 5113 raft_consensus.cc:740] T d1f7e5e193714b738cc629970ae5c054 P ecdb77fe301541a4a0f652e78308dc80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecdb77fe301541a4a0f652e78308dc80, State: Initialized, Role: FOLLOWER
I20260429 14:03:46.875767 5113 consensus_queue.cc:260] T d1f7e5e193714b738cc629970ae5c054 P ecdb77fe301541a4a0f652e78308dc80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:46.875890 5113 ts_tablet_manager.cc:1434] T d1f7e5e193714b738cc629970ae5c054 P ecdb77fe301541a4a0f652e78308dc80: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 14:03:47.066380 5121 raft_consensus.cc:493] T d1f7e5e193714b738cc629970ae5c054 P f50d561a4167470b961a3f7dbeb6ef3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:03:47.066509 5121 raft_consensus.cc:515] T d1f7e5e193714b738cc629970ae5c054 P f50d561a4167470b961a3f7dbeb6ef3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:47.066744 5121 leader_election.cc:290] T d1f7e5e193714b738cc629970ae5c054 P f50d561a4167470b961a3f7dbeb6ef3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecdb77fe301541a4a0f652e78308dc80 (127.2.197.193:42969), 12d3554b22244952a8cb77fa76ad9e38 (127.2.197.195:40127)
I20260429 14:03:47.066941 4758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f7e5e193714b738cc629970ae5c054" candidate_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecdb77fe301541a4a0f652e78308dc80" is_pre_election: true
I20260429 14:03:47.067030 4758 raft_consensus.cc:2468] T d1f7e5e193714b738cc629970ae5c054 P ecdb77fe301541a4a0f652e78308dc80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f50d561a4167470b961a3f7dbeb6ef3e in term 0.
I20260429 14:03:47.067041 5042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f7e5e193714b738cc629970ae5c054" candidate_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d3554b22244952a8cb77fa76ad9e38" is_pre_election: true
I20260429 14:03:47.067209 5042 raft_consensus.cc:2468] T d1f7e5e193714b738cc629970ae5c054 P 12d3554b22244952a8cb77fa76ad9e38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f50d561a4167470b961a3f7dbeb6ef3e in term 0.
I20260429 14:03:47.067214 4832 leader_election.cc:304] T d1f7e5e193714b738cc629970ae5c054 P f50d561a4167470b961a3f7dbeb6ef3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ecdb77fe301541a4a0f652e78308dc80, f50d561a4167470b961a3f7dbeb6ef3e; no voters:
I20260429 14:03:47.067312 5121 raft_consensus.cc:2804] T d1f7e5e193714b738cc629970ae5c054 P f50d561a4167470b961a3f7dbeb6ef3e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 14:03:47.067400 5121 raft_consensus.cc:493] T d1f7e5e193714b738cc629970ae5c054 P f50d561a4167470b961a3f7dbeb6ef3e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 14:03:47.067436 5121 raft_consensus.cc:3060] T d1f7e5e193714b738cc629970ae5c054 P f50d561a4167470b961a3f7dbeb6ef3e [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:03:47.068176 5121 raft_consensus.cc:515] T d1f7e5e193714b738cc629970ae5c054 P f50d561a4167470b961a3f7dbeb6ef3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:47.068336 5121 leader_election.cc:290] T d1f7e5e193714b738cc629970ae5c054 P f50d561a4167470b961a3f7dbeb6ef3e [CANDIDATE]: Term 1 election: Requested vote from peers ecdb77fe301541a4a0f652e78308dc80 (127.2.197.193:42969), 12d3554b22244952a8cb77fa76ad9e38 (127.2.197.195:40127)
I20260429 14:03:47.068627 4758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f7e5e193714b738cc629970ae5c054" candidate_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecdb77fe301541a4a0f652e78308dc80"
I20260429 14:03:47.068636 5042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f7e5e193714b738cc629970ae5c054" candidate_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d3554b22244952a8cb77fa76ad9e38"
I20260429 14:03:47.068720 5042 raft_consensus.cc:3060] T d1f7e5e193714b738cc629970ae5c054 P 12d3554b22244952a8cb77fa76ad9e38 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:03:47.068723 4758 raft_consensus.cc:3060] T d1f7e5e193714b738cc629970ae5c054 P ecdb77fe301541a4a0f652e78308dc80 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:03:47.069314 5042 raft_consensus.cc:2468] T d1f7e5e193714b738cc629970ae5c054 P 12d3554b22244952a8cb77fa76ad9e38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f50d561a4167470b961a3f7dbeb6ef3e in term 1.
I20260429 14:03:47.069322 4758 raft_consensus.cc:2468] T d1f7e5e193714b738cc629970ae5c054 P ecdb77fe301541a4a0f652e78308dc80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f50d561a4167470b961a3f7dbeb6ef3e in term 1.
I20260429 14:03:47.069499 4832 leader_election.cc:304] T d1f7e5e193714b738cc629970ae5c054 P f50d561a4167470b961a3f7dbeb6ef3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ecdb77fe301541a4a0f652e78308dc80, f50d561a4167470b961a3f7dbeb6ef3e; no voters:
I20260429 14:03:47.069646 5121 raft_consensus.cc:2804] T d1f7e5e193714b738cc629970ae5c054 P f50d561a4167470b961a3f7dbeb6ef3e [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:03:47.069712 5121 raft_consensus.cc:697] T d1f7e5e193714b738cc629970ae5c054 P f50d561a4167470b961a3f7dbeb6ef3e [term 1 LEADER]: Becoming Leader. State: Replica: f50d561a4167470b961a3f7dbeb6ef3e, State: Running, Role: LEADER
I20260429 14:03:47.069772 5121 consensus_queue.cc:237] T d1f7e5e193714b738cc629970ae5c054 P f50d561a4167470b961a3f7dbeb6ef3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:47.070657 4597 catalog_manager.cc:5671] T d1f7e5e193714b738cc629970ae5c054 P f50d561a4167470b961a3f7dbeb6ef3e reported cstate change: term changed from 0 to 1, leader changed from <none> to f50d561a4167470b961a3f7dbeb6ef3e (127.2.197.194). New cstate: current_term: 1 leader_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } health_report { overall_health: UNKNOWN } } }
I20260429 14:03:47.099445 4758 raft_consensus.cc:1275] T d1f7e5e193714b738cc629970ae5c054 P ecdb77fe301541a4a0f652e78308dc80 [term 1 FOLLOWER]: Refusing update from remote peer f50d561a4167470b961a3f7dbeb6ef3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:03:47.099480 5042 raft_consensus.cc:1275] T d1f7e5e193714b738cc629970ae5c054 P 12d3554b22244952a8cb77fa76ad9e38 [term 1 FOLLOWER]: Refusing update from remote peer f50d561a4167470b961a3f7dbeb6ef3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:03:47.099704 5118 consensus_queue.cc:1048] T d1f7e5e193714b738cc629970ae5c054 P f50d561a4167470b961a3f7dbeb6ef3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:03:47.099862 5153 consensus_queue.cc:1048] T d1f7e5e193714b738cc629970ae5c054 P f50d561a4167470b961a3f7dbeb6ef3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260429 14:03:47.104496 5139 tablet_replica.cc:1307] Aborted: operation has been aborted: cancelling pending write operations
I20260429 14:03:48.626714 4599 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:03:48.626855 4599 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:03:48.628137 4599 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 1aeac3f977874eb79cf735db9587a3eb: Sending DeleteTablet for 3 replicas of tablet d1f7e5e193714b738cc629970ae5c054
I20260429 14:03:48.628625 5021 tablet_service.cc:1558] Processing DeleteTablet for tablet d1f7e5e193714b738cc629970ae5c054 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:03:48 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:49430
I20260429 14:03:48.628634 4880 tablet_service.cc:1558] Processing DeleteTablet for tablet d1f7e5e193714b738cc629970ae5c054 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:03:48 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:45904
I20260429 14:03:48.628669 4738 tablet_service.cc:1558] Processing DeleteTablet for tablet d1f7e5e193714b738cc629970ae5c054 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:03:48 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:52900
I20260429 14:03:48.628868 5187 tablet_replica.cc:333] T d1f7e5e193714b738cc629970ae5c054 P 12d3554b22244952a8cb77fa76ad9e38: stopping tablet replica
I20260429 14:03:48.628927 5187 raft_consensus.cc:2243] T d1f7e5e193714b738cc629970ae5c054 P 12d3554b22244952a8cb77fa76ad9e38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:03:48.628978 5187 raft_consensus.cc:2272] T d1f7e5e193714b738cc629970ae5c054 P 12d3554b22244952a8cb77fa76ad9e38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:03:48.629266 5187 ts_tablet_manager.cc:1916] T d1f7e5e193714b738cc629970ae5c054 P 12d3554b22244952a8cb77fa76ad9e38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:03:48.631325 5188 tablet_replica.cc:333] T d1f7e5e193714b738cc629970ae5c054 P f50d561a4167470b961a3f7dbeb6ef3e: stopping tablet replica
I20260429 14:03:48.631397 5188 raft_consensus.cc:2243] T d1f7e5e193714b738cc629970ae5c054 P f50d561a4167470b961a3f7dbeb6ef3e [term 1 LEADER]: Raft consensus shutting down.
I20260429 14:03:48.631484 5188 raft_consensus.cc:2272] T d1f7e5e193714b738cc629970ae5c054 P f50d561a4167470b961a3f7dbeb6ef3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:03:48.631511 5187 ts_tablet_manager.cc:1929] T d1f7e5e193714b738cc629970ae5c054 P 12d3554b22244952a8cb77fa76ad9e38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260429 14:03:48.631615 5187 log.cc:1199] T d1f7e5e193714b738cc629970ae5c054 P 12d3554b22244952a8cb77fa76ad9e38: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-2/wal/wals/d1f7e5e193714b738cc629970ae5c054
I20260429 14:03:48.631753 5188 ts_tablet_manager.cc:1916] T d1f7e5e193714b738cc629970ae5c054 P f50d561a4167470b961a3f7dbeb6ef3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:03:48.631903 5187 ts_tablet_manager.cc:1950] T d1f7e5e193714b738cc629970ae5c054 P 12d3554b22244952a8cb77fa76ad9e38: Deleting consensus metadata
I20260429 14:03:48.632241 4586 catalog_manager.cc:5002] TS 12d3554b22244952a8cb77fa76ad9e38 (127.2.197.195:40127): tablet d1f7e5e193714b738cc629970ae5c054 (table test-table [id=66ec6fa1d6164ff3a92943d60d93e28c]) successfully deleted
I20260429 14:03:48.633249 5189 tablet_replica.cc:333] T d1f7e5e193714b738cc629970ae5c054 P ecdb77fe301541a4a0f652e78308dc80: stopping tablet replica
I20260429 14:03:48.633297 5188 ts_tablet_manager.cc:1929] T d1f7e5e193714b738cc629970ae5c054 P f50d561a4167470b961a3f7dbeb6ef3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260429 14:03:48.633352 5188 log.cc:1199] T d1f7e5e193714b738cc629970ae5c054 P f50d561a4167470b961a3f7dbeb6ef3e: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-1/wal/wals/d1f7e5e193714b738cc629970ae5c054
I20260429 14:03:48.633340 5189 raft_consensus.cc:2243] T d1f7e5e193714b738cc629970ae5c054 P ecdb77fe301541a4a0f652e78308dc80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:03:48.633389 5189 raft_consensus.cc:2272] T d1f7e5e193714b738cc629970ae5c054 P ecdb77fe301541a4a0f652e78308dc80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:03:48.633612 5188 ts_tablet_manager.cc:1950] T d1f7e5e193714b738cc629970ae5c054 P f50d561a4167470b961a3f7dbeb6ef3e: Deleting consensus metadata
I20260429 14:03:48.633672 5189 ts_tablet_manager.cc:1916] T d1f7e5e193714b738cc629970ae5c054 P ecdb77fe301541a4a0f652e78308dc80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:03:48.633937 4584 catalog_manager.cc:5002] TS f50d561a4167470b961a3f7dbeb6ef3e (127.2.197.194:37203): tablet d1f7e5e193714b738cc629970ae5c054 (table test-table [id=66ec6fa1d6164ff3a92943d60d93e28c]) successfully deleted
I20260429 14:03:48.635363 5189 ts_tablet_manager.cc:1929] T d1f7e5e193714b738cc629970ae5c054 P ecdb77fe301541a4a0f652e78308dc80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260429 14:03:48.635465 5189 log.cc:1199] T d1f7e5e193714b738cc629970ae5c054 P ecdb77fe301541a4a0f652e78308dc80: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-0/wal/wals/d1f7e5e193714b738cc629970ae5c054
I20260429 14:03:48.635689 5189 ts_tablet_manager.cc:1950] T d1f7e5e193714b738cc629970ae5c054 P ecdb77fe301541a4a0f652e78308dc80: Deleting consensus metadata
I20260429 14:03:48.636109 4584 catalog_manager.cc:5002] TS ecdb77fe301541a4a0f652e78308dc80 (127.2.197.193:42969): tablet d1f7e5e193714b738cc629970ae5c054 (table test-table [id=66ec6fa1d6164ff3a92943d60d93e28c]) successfully deleted
Apr 29 14:03:48 dist-test-slave-8n77 krb5kdc[4551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471428, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Apr 29 14:03:48 dist-test-slave-8n77 krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471428, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:03:48.656575 5197 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:48.645496 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48574 (local address 127.2.197.254:36237)
0429 14:03:48.645694 (+ 198us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:48.645696 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:48.645719 (+ 23us) server_negotiation.cc:408] Connection header received
0429 14:03:48.645754 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:48.645755 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:48.645765 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:48.645836 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:48.646578 (+ 742us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:48.647099 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:48.647701 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:48.647905 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:48.653882 (+ 5977us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:48.653888 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:48.653888 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:48.653896 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:48.655475 (+ 1579us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:48.655850 (+ 375us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:48.655852 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:48.655852 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:48.655875 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:48.656018 (+ 143us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:48.656019 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:48.656019 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:48.656117 (+ 98us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:48.656196 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:48.656308 (+ 112us) server_negotiation.cc:300] Negotiation successful
0429 14:03:48.656392 (+ 84us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":119,"threads_started":1}
I20260429 14:03:48.658224 4599 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 14:03:48.658452 4599 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 14:03:48.660065 4738 tablet_service.cc:1511] Processing CreateTablet for tablet 4fb846d38ab84888ab0562923f652b5c (DEFAULT_TABLE table=test-table [id=e5ece411d7c7449f86036767e6a358cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:03:48.660106 4880 tablet_service.cc:1511] Processing CreateTablet for tablet 4fb846d38ab84888ab0562923f652b5c (DEFAULT_TABLE table=test-table [id=e5ece411d7c7449f86036767e6a358cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:03:48.660199 4738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fb846d38ab84888ab0562923f652b5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:03:48.660208 4880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fb846d38ab84888ab0562923f652b5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:03:48.661566 5198 tablet_bootstrap.cc:492] T 4fb846d38ab84888ab0562923f652b5c P ecdb77fe301541a4a0f652e78308dc80: Bootstrap starting.
I20260429 14:03:48.661587 5021 tablet_service.cc:1511] Processing CreateTablet for tablet 4fb846d38ab84888ab0562923f652b5c (DEFAULT_TABLE table=test-table [id=e5ece411d7c7449f86036767e6a358cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:03:48.661695 5021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fb846d38ab84888ab0562923f652b5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:03:48.661974 5198 tablet_bootstrap.cc:654] T 4fb846d38ab84888ab0562923f652b5c P ecdb77fe301541a4a0f652e78308dc80: Neither blocks nor log segments found. Creating new log.
I20260429 14:03:48.662777 5198 tablet_bootstrap.cc:492] T 4fb846d38ab84888ab0562923f652b5c P ecdb77fe301541a4a0f652e78308dc80: No bootstrap required, opened a new log
I20260429 14:03:48.662839 5198 ts_tablet_manager.cc:1403] T 4fb846d38ab84888ab0562923f652b5c P ecdb77fe301541a4a0f652e78308dc80: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260429 14:03:48.662963 5198 raft_consensus.cc:359] T 4fb846d38ab84888ab0562923f652b5c P ecdb77fe301541a4a0f652e78308dc80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:48.663039 5198 raft_consensus.cc:385] T 4fb846d38ab84888ab0562923f652b5c P ecdb77fe301541a4a0f652e78308dc80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:03:48.663060 5198 raft_consensus.cc:740] T 4fb846d38ab84888ab0562923f652b5c P ecdb77fe301541a4a0f652e78308dc80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecdb77fe301541a4a0f652e78308dc80, State: Initialized, Role: FOLLOWER
I20260429 14:03:48.663131 5198 consensus_queue.cc:260] T 4fb846d38ab84888ab0562923f652b5c P ecdb77fe301541a4a0f652e78308dc80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:48.663395 5201 tablet_bootstrap.cc:492] T 4fb846d38ab84888ab0562923f652b5c P 12d3554b22244952a8cb77fa76ad9e38: Bootstrap starting.
I20260429 14:03:48.663488 5198 ts_tablet_manager.cc:1434] T 4fb846d38ab84888ab0562923f652b5c P ecdb77fe301541a4a0f652e78308dc80: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 14:03:48.663785 5201 tablet_bootstrap.cc:654] T 4fb846d38ab84888ab0562923f652b5c P 12d3554b22244952a8cb77fa76ad9e38: Neither blocks nor log segments found. Creating new log.
I20260429 14:03:48.664238 5199 tablet_bootstrap.cc:492] T 4fb846d38ab84888ab0562923f652b5c P f50d561a4167470b961a3f7dbeb6ef3e: Bootstrap starting.
I20260429 14:03:48.664512 5201 tablet_bootstrap.cc:492] T 4fb846d38ab84888ab0562923f652b5c P 12d3554b22244952a8cb77fa76ad9e38: No bootstrap required, opened a new log
I20260429 14:03:48.664561 5201 ts_tablet_manager.cc:1403] T 4fb846d38ab84888ab0562923f652b5c P 12d3554b22244952a8cb77fa76ad9e38: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260429 14:03:48.664685 5199 tablet_bootstrap.cc:654] T 4fb846d38ab84888ab0562923f652b5c P f50d561a4167470b961a3f7dbeb6ef3e: Neither blocks nor log segments found. Creating new log.
I20260429 14:03:48.664685 5201 raft_consensus.cc:359] T 4fb846d38ab84888ab0562923f652b5c P 12d3554b22244952a8cb77fa76ad9e38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:48.664732 5201 raft_consensus.cc:385] T 4fb846d38ab84888ab0562923f652b5c P 12d3554b22244952a8cb77fa76ad9e38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:03:48.664746 5201 raft_consensus.cc:740] T 4fb846d38ab84888ab0562923f652b5c P 12d3554b22244952a8cb77fa76ad9e38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12d3554b22244952a8cb77fa76ad9e38, State: Initialized, Role: FOLLOWER
I20260429 14:03:48.664842 5201 consensus_queue.cc:260] T 4fb846d38ab84888ab0562923f652b5c P 12d3554b22244952a8cb77fa76ad9e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:48.664947 5201 ts_tablet_manager.cc:1434] T 4fb846d38ab84888ab0562923f652b5c P 12d3554b22244952a8cb77fa76ad9e38: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:48.665287 5199 tablet_bootstrap.cc:492] T 4fb846d38ab84888ab0562923f652b5c P f50d561a4167470b961a3f7dbeb6ef3e: No bootstrap required, opened a new log
I20260429 14:03:48.665340 5199 ts_tablet_manager.cc:1403] T 4fb846d38ab84888ab0562923f652b5c P f50d561a4167470b961a3f7dbeb6ef3e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260429 14:03:48.665494 5199 raft_consensus.cc:359] T 4fb846d38ab84888ab0562923f652b5c P f50d561a4167470b961a3f7dbeb6ef3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:48.665592 5199 raft_consensus.cc:385] T 4fb846d38ab84888ab0562923f652b5c P f50d561a4167470b961a3f7dbeb6ef3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:03:48.665623 5199 raft_consensus.cc:740] T 4fb846d38ab84888ab0562923f652b5c P f50d561a4167470b961a3f7dbeb6ef3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f50d561a4167470b961a3f7dbeb6ef3e, State: Initialized, Role: FOLLOWER
I20260429 14:03:48.665673 5199 consensus_queue.cc:260] T 4fb846d38ab84888ab0562923f652b5c P f50d561a4167470b961a3f7dbeb6ef3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:48.665827 5199 ts_tablet_manager.cc:1434] T 4fb846d38ab84888ab0562923f652b5c P f50d561a4167470b961a3f7dbeb6ef3e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260429 14:03:48.714082 5181 raft_consensus.cc:493] T 4fb846d38ab84888ab0562923f652b5c P f50d561a4167470b961a3f7dbeb6ef3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:03:48.714234 5181 raft_consensus.cc:515] T 4fb846d38ab84888ab0562923f652b5c P f50d561a4167470b961a3f7dbeb6ef3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:48.714465 5181 leader_election.cc:290] T 4fb846d38ab84888ab0562923f652b5c P f50d561a4167470b961a3f7dbeb6ef3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecdb77fe301541a4a0f652e78308dc80 (127.2.197.193:42969), 12d3554b22244952a8cb77fa76ad9e38 (127.2.197.195:40127)
I20260429 14:03:48.714740 5042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4fb846d38ab84888ab0562923f652b5c" candidate_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d3554b22244952a8cb77fa76ad9e38" is_pre_election: true
I20260429 14:03:48.714740 4758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4fb846d38ab84888ab0562923f652b5c" candidate_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecdb77fe301541a4a0f652e78308dc80" is_pre_election: true
I20260429 14:03:48.714841 4758 raft_consensus.cc:2468] T 4fb846d38ab84888ab0562923f652b5c P ecdb77fe301541a4a0f652e78308dc80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f50d561a4167470b961a3f7dbeb6ef3e in term 0.
I20260429 14:03:48.714856 5042 raft_consensus.cc:2468] T 4fb846d38ab84888ab0562923f652b5c P 12d3554b22244952a8cb77fa76ad9e38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f50d561a4167470b961a3f7dbeb6ef3e in term 0.
I20260429 14:03:48.715040 4834 leader_election.cc:304] T 4fb846d38ab84888ab0562923f652b5c P f50d561a4167470b961a3f7dbeb6ef3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12d3554b22244952a8cb77fa76ad9e38, f50d561a4167470b961a3f7dbeb6ef3e; no voters:
I20260429 14:03:48.715216 5181 raft_consensus.cc:2804] T 4fb846d38ab84888ab0562923f652b5c P f50d561a4167470b961a3f7dbeb6ef3e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 14:03:48.715276 5181 raft_consensus.cc:493] T 4fb846d38ab84888ab0562923f652b5c P f50d561a4167470b961a3f7dbeb6ef3e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 14:03:48.715298 5181 raft_consensus.cc:3060] T 4fb846d38ab84888ab0562923f652b5c P f50d561a4167470b961a3f7dbeb6ef3e [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:03:48.716023 5181 raft_consensus.cc:515] T 4fb846d38ab84888ab0562923f652b5c P f50d561a4167470b961a3f7dbeb6ef3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:48.716166 5181 leader_election.cc:290] T 4fb846d38ab84888ab0562923f652b5c P f50d561a4167470b961a3f7dbeb6ef3e [CANDIDATE]: Term 1 election: Requested vote from peers ecdb77fe301541a4a0f652e78308dc80 (127.2.197.193:42969), 12d3554b22244952a8cb77fa76ad9e38 (127.2.197.195:40127)
I20260429 14:03:48.716341 4758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4fb846d38ab84888ab0562923f652b5c" candidate_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecdb77fe301541a4a0f652e78308dc80"
I20260429 14:03:48.716336 5042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4fb846d38ab84888ab0562923f652b5c" candidate_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d3554b22244952a8cb77fa76ad9e38"
I20260429 14:03:48.716423 5042 raft_consensus.cc:3060] T 4fb846d38ab84888ab0562923f652b5c P 12d3554b22244952a8cb77fa76ad9e38 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:03:48.716424 4758 raft_consensus.cc:3060] T 4fb846d38ab84888ab0562923f652b5c P ecdb77fe301541a4a0f652e78308dc80 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:03:48.717072 4758 raft_consensus.cc:2468] T 4fb846d38ab84888ab0562923f652b5c P ecdb77fe301541a4a0f652e78308dc80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f50d561a4167470b961a3f7dbeb6ef3e in term 1.
I20260429 14:03:48.717209 5042 raft_consensus.cc:2468] T 4fb846d38ab84888ab0562923f652b5c P 12d3554b22244952a8cb77fa76ad9e38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f50d561a4167470b961a3f7dbeb6ef3e in term 1.
I20260429 14:03:48.717284 4832 leader_election.cc:304] T 4fb846d38ab84888ab0562923f652b5c P f50d561a4167470b961a3f7dbeb6ef3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ecdb77fe301541a4a0f652e78308dc80, f50d561a4167470b961a3f7dbeb6ef3e; no voters:
I20260429 14:03:48.717378 5181 raft_consensus.cc:2804] T 4fb846d38ab84888ab0562923f652b5c P f50d561a4167470b961a3f7dbeb6ef3e [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:03:48.717448 5181 raft_consensus.cc:697] T 4fb846d38ab84888ab0562923f652b5c P f50d561a4167470b961a3f7dbeb6ef3e [term 1 LEADER]: Becoming Leader. State: Replica: f50d561a4167470b961a3f7dbeb6ef3e, State: Running, Role: LEADER
I20260429 14:03:48.717505 5181 consensus_queue.cc:237] T 4fb846d38ab84888ab0562923f652b5c P f50d561a4167470b961a3f7dbeb6ef3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:48.718169 4597 catalog_manager.cc:5671] T 4fb846d38ab84888ab0562923f652b5c P f50d561a4167470b961a3f7dbeb6ef3e reported cstate change: term changed from 0 to 1, leader changed from <none> to f50d561a4167470b961a3f7dbeb6ef3e (127.2.197.194). New cstate: current_term: 1 leader_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } health_report { overall_health: UNKNOWN } } }
I20260429 14:03:48.727146 5207 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:48.724462 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54420 (local address 127.2.197.194:37203)
0429 14:03:48.724699 (+ 237us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:48.724701 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:48.724712 (+ 11us) server_negotiation.cc:408] Connection header received
0429 14:03:48.724731 (+ 19us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:48.724732 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:48.724748 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:48.724830 (+ 82us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:03:48.725229 (+ 399us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:48.725874 (+ 645us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:48.726532 (+ 658us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:48.726675 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:48.726695 (+ 20us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:03:48.726717 (+ 22us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:03:48.726923 (+ 206us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:48.727001 (+ 78us) server_negotiation.cc:300] Negotiation successful
0429 14:03:48.727025 (+ 24us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":142,"threads_started":1}
I20260429 14:03:48.729882 5042 raft_consensus.cc:1275] T 4fb846d38ab84888ab0562923f652b5c P 12d3554b22244952a8cb77fa76ad9e38 [term 1 FOLLOWER]: Refusing update from remote peer f50d561a4167470b961a3f7dbeb6ef3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:03:48.730015 4758 raft_consensus.cc:1275] T 4fb846d38ab84888ab0562923f652b5c P ecdb77fe301541a4a0f652e78308dc80 [term 1 FOLLOWER]: Refusing update from remote peer f50d561a4167470b961a3f7dbeb6ef3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:03:48.730108 5181 consensus_queue.cc:1048] T 4fb846d38ab84888ab0562923f652b5c P f50d561a4167470b961a3f7dbeb6ef3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:03:48.730265 5178 consensus_queue.cc:1048] T 4fb846d38ab84888ab0562923f652b5c P f50d561a4167470b961a3f7dbeb6ef3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:03:48.740708 4597 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:03:48.740825 4597 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:03:48.741488 4597 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 1aeac3f977874eb79cf735db9587a3eb: Sending DeleteTablet for 3 replicas of tablet 4fb846d38ab84888ab0562923f652b5c
I20260429 14:03:48.741873 4880 tablet_service.cc:1558] Processing DeleteTablet for tablet 4fb846d38ab84888ab0562923f652b5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:03:48 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:45904
I20260429 14:03:48.741876 4738 tablet_service.cc:1558] Processing DeleteTablet for tablet 4fb846d38ab84888ab0562923f652b5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:03:48 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:52900
I20260429 14:03:48.741894 5021 tablet_service.cc:1558] Processing DeleteTablet for tablet 4fb846d38ab84888ab0562923f652b5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:03:48 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:49430
I20260429 14:03:48.741998 5189 tablet_replica.cc:333] T 4fb846d38ab84888ab0562923f652b5c P ecdb77fe301541a4a0f652e78308dc80: stopping tablet replica
I20260429 14:03:48.741998 5187 tablet_replica.cc:333] T 4fb846d38ab84888ab0562923f652b5c P 12d3554b22244952a8cb77fa76ad9e38: stopping tablet replica
I20260429 14:03:48.741997 5188 tablet_replica.cc:333] T 4fb846d38ab84888ab0562923f652b5c P f50d561a4167470b961a3f7dbeb6ef3e: stopping tablet replica
I20260429 14:03:48.742056 5187 raft_consensus.cc:2243] T 4fb846d38ab84888ab0562923f652b5c P 12d3554b22244952a8cb77fa76ad9e38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:03:48.742056 5189 raft_consensus.cc:2243] T 4fb846d38ab84888ab0562923f652b5c P ecdb77fe301541a4a0f652e78308dc80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:03:48.742070 5188 raft_consensus.cc:2243] T 4fb846d38ab84888ab0562923f652b5c P f50d561a4167470b961a3f7dbeb6ef3e [term 1 LEADER]: Raft consensus shutting down.
I20260429 14:03:48.742106 5187 raft_consensus.cc:2272] T 4fb846d38ab84888ab0562923f652b5c P 12d3554b22244952a8cb77fa76ad9e38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:03:48.742105 5189 raft_consensus.cc:2272] T 4fb846d38ab84888ab0562923f652b5c P ecdb77fe301541a4a0f652e78308dc80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:03:48.742129 5188 raft_consensus.cc:2272] T 4fb846d38ab84888ab0562923f652b5c P f50d561a4167470b961a3f7dbeb6ef3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:03:48.742625 5188 ts_tablet_manager.cc:1916] T 4fb846d38ab84888ab0562923f652b5c P f50d561a4167470b961a3f7dbeb6ef3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:03:48.742569 4597 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 14:03:48.742715 4597 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 14:03:48.743323 5189 ts_tablet_manager.cc:1916] T 4fb846d38ab84888ab0562923f652b5c P ecdb77fe301541a4a0f652e78308dc80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:03:48.743664 5187 ts_tablet_manager.cc:1916] T 4fb846d38ab84888ab0562923f652b5c P 12d3554b22244952a8cb77fa76ad9e38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:03:48.745292 5189 ts_tablet_manager.cc:1929] T 4fb846d38ab84888ab0562923f652b5c P ecdb77fe301541a4a0f652e78308dc80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 14:03:48.745364 5189 log.cc:1199] T 4fb846d38ab84888ab0562923f652b5c P ecdb77fe301541a4a0f652e78308dc80: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-0/wal/wals/4fb846d38ab84888ab0562923f652b5c
I20260429 14:03:48.745330 5188 ts_tablet_manager.cc:1929] T 4fb846d38ab84888ab0562923f652b5c P f50d561a4167470b961a3f7dbeb6ef3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 14:03:48.745426 5188 log.cc:1199] T 4fb846d38ab84888ab0562923f652b5c P f50d561a4167470b961a3f7dbeb6ef3e: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-1/wal/wals/4fb846d38ab84888ab0562923f652b5c
I20260429 14:03:48.745612 5189 ts_tablet_manager.cc:1950] T 4fb846d38ab84888ab0562923f652b5c P ecdb77fe301541a4a0f652e78308dc80: Deleting consensus metadata
I20260429 14:03:48.745838 5188 ts_tablet_manager.cc:1950] T 4fb846d38ab84888ab0562923f652b5c P f50d561a4167470b961a3f7dbeb6ef3e: Deleting consensus metadata
I20260429 14:03:48.745966 5187 ts_tablet_manager.cc:1929] T 4fb846d38ab84888ab0562923f652b5c P 12d3554b22244952a8cb77fa76ad9e38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 14:03:48.746018 5187 log.cc:1199] T 4fb846d38ab84888ab0562923f652b5c P 12d3554b22244952a8cb77fa76ad9e38: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-2/wal/wals/4fb846d38ab84888ab0562923f652b5c
I20260429 14:03:48.746205 4584 catalog_manager.cc:5002] TS f50d561a4167470b961a3f7dbeb6ef3e (127.2.197.194:37203): tablet 4fb846d38ab84888ab0562923f652b5c (table test-table [id=e5ece411d7c7449f86036767e6a358cd]) successfully deleted
I20260429 14:03:48.746225 5187 ts_tablet_manager.cc:1950] T 4fb846d38ab84888ab0562923f652b5c P 12d3554b22244952a8cb77fa76ad9e38: Deleting consensus metadata
I20260429 14:03:48.746315 4584 catalog_manager.cc:5002] TS ecdb77fe301541a4a0f652e78308dc80 (127.2.197.193:42969): tablet 4fb846d38ab84888ab0562923f652b5c (table test-table [id=e5ece411d7c7449f86036767e6a358cd]) successfully deleted
I20260429 14:03:48.746603 4738 tablet_service.cc:1511] Processing CreateTablet for tablet c0837da4be774518841acb6daa85207d (DEFAULT_TABLE table=test-table [id=d6f692f065b54d4aaf68bff9f6d13f36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:03:48.746652 4586 catalog_manager.cc:5002] TS 12d3554b22244952a8cb77fa76ad9e38 (127.2.197.195:40127): tablet 4fb846d38ab84888ab0562923f652b5c (table test-table [id=e5ece411d7c7449f86036767e6a358cd]) successfully deleted
I20260429 14:03:48.746721 4738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0837da4be774518841acb6daa85207d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:03:48.746750 4880 tablet_service.cc:1511] Processing CreateTablet for tablet c0837da4be774518841acb6daa85207d (DEFAULT_TABLE table=test-table [id=d6f692f065b54d4aaf68bff9f6d13f36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:03:48.746932 5021 tablet_service.cc:1511] Processing CreateTablet for tablet c0837da4be774518841acb6daa85207d (DEFAULT_TABLE table=test-table [id=d6f692f065b54d4aaf68bff9f6d13f36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:03:48.747080 5021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0837da4be774518841acb6daa85207d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:03:48.747116 4880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0837da4be774518841acb6daa85207d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:03:48.748659 5198 tablet_bootstrap.cc:492] T c0837da4be774518841acb6daa85207d P ecdb77fe301541a4a0f652e78308dc80: Bootstrap starting.
I20260429 14:03:48.748672 5201 tablet_bootstrap.cc:492] T c0837da4be774518841acb6daa85207d P 12d3554b22244952a8cb77fa76ad9e38: Bootstrap starting.
I20260429 14:03:48.749044 5201 tablet_bootstrap.cc:654] T c0837da4be774518841acb6daa85207d P 12d3554b22244952a8cb77fa76ad9e38: Neither blocks nor log segments found. Creating new log.
I20260429 14:03:48.749087 5199 tablet_bootstrap.cc:492] T c0837da4be774518841acb6daa85207d P f50d561a4167470b961a3f7dbeb6ef3e: Bootstrap starting.
I20260429 14:03:48.749225 5198 tablet_bootstrap.cc:654] T c0837da4be774518841acb6daa85207d P ecdb77fe301541a4a0f652e78308dc80: Neither blocks nor log segments found. Creating new log.
I20260429 14:03:48.749473 5199 tablet_bootstrap.cc:654] T c0837da4be774518841acb6daa85207d P f50d561a4167470b961a3f7dbeb6ef3e: Neither blocks nor log segments found. Creating new log.
I20260429 14:03:48.749933 5201 tablet_bootstrap.cc:492] T c0837da4be774518841acb6daa85207d P 12d3554b22244952a8cb77fa76ad9e38: No bootstrap required, opened a new log
I20260429 14:03:48.749953 5198 tablet_bootstrap.cc:492] T c0837da4be774518841acb6daa85207d P ecdb77fe301541a4a0f652e78308dc80: No bootstrap required, opened a new log
I20260429 14:03:48.749979 5201 ts_tablet_manager.cc:1403] T c0837da4be774518841acb6daa85207d P 12d3554b22244952a8cb77fa76ad9e38: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260429 14:03:48.749998 5198 ts_tablet_manager.cc:1403] T c0837da4be774518841acb6daa85207d P ecdb77fe301541a4a0f652e78308dc80: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260429 14:03:48.750169 5201 raft_consensus.cc:359] T c0837da4be774518841acb6daa85207d P 12d3554b22244952a8cb77fa76ad9e38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:48.750187 5198 raft_consensus.cc:359] T c0837da4be774518841acb6daa85207d P ecdb77fe301541a4a0f652e78308dc80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:48.750232 5199 tablet_bootstrap.cc:492] T c0837da4be774518841acb6daa85207d P f50d561a4167470b961a3f7dbeb6ef3e: No bootstrap required, opened a new log
I20260429 14:03:48.750252 5201 raft_consensus.cc:385] T c0837da4be774518841acb6daa85207d P 12d3554b22244952a8cb77fa76ad9e38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:03:48.750281 5198 raft_consensus.cc:385] T c0837da4be774518841acb6daa85207d P ecdb77fe301541a4a0f652e78308dc80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:03:48.750304 5201 raft_consensus.cc:740] T c0837da4be774518841acb6daa85207d P 12d3554b22244952a8cb77fa76ad9e38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12d3554b22244952a8cb77fa76ad9e38, State: Initialized, Role: FOLLOWER
I20260429 14:03:48.750315 5199 ts_tablet_manager.cc:1403] T c0837da4be774518841acb6daa85207d P f50d561a4167470b961a3f7dbeb6ef3e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260429 14:03:48.750326 5198 raft_consensus.cc:740] T c0837da4be774518841acb6daa85207d P ecdb77fe301541a4a0f652e78308dc80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecdb77fe301541a4a0f652e78308dc80, State: Initialized, Role: FOLLOWER
I20260429 14:03:48.750358 5201 consensus_queue.cc:260] T c0837da4be774518841acb6daa85207d P 12d3554b22244952a8cb77fa76ad9e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:48.750377 5198 consensus_queue.cc:260] T c0837da4be774518841acb6daa85207d P ecdb77fe301541a4a0f652e78308dc80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:48.750455 5201 ts_tablet_manager.cc:1434] T c0837da4be774518841acb6daa85207d P 12d3554b22244952a8cb77fa76ad9e38: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260429 14:03:48.750461 5198 ts_tablet_manager.cc:1434] T c0837da4be774518841acb6daa85207d P ecdb77fe301541a4a0f652e78308dc80: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260429 14:03:48.750438 5199 raft_consensus.cc:359] T c0837da4be774518841acb6daa85207d P f50d561a4167470b961a3f7dbeb6ef3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:48.750512 5199 raft_consensus.cc:385] T c0837da4be774518841acb6daa85207d P f50d561a4167470b961a3f7dbeb6ef3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:03:48.750538 5199 raft_consensus.cc:740] T c0837da4be774518841acb6daa85207d P f50d561a4167470b961a3f7dbeb6ef3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f50d561a4167470b961a3f7dbeb6ef3e, State: Initialized, Role: FOLLOWER
I20260429 14:03:48.750582 5199 consensus_queue.cc:260] T c0837da4be774518841acb6daa85207d P f50d561a4167470b961a3f7dbeb6ef3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:48.750672 5199 ts_tablet_manager.cc:1434] T c0837da4be774518841acb6daa85207d P f50d561a4167470b961a3f7dbeb6ef3e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:49.056507 5202 raft_consensus.cc:493] T c0837da4be774518841acb6daa85207d P ecdb77fe301541a4a0f652e78308dc80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:03:49.056615 5202 raft_consensus.cc:515] T c0837da4be774518841acb6daa85207d P ecdb77fe301541a4a0f652e78308dc80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:49.056942 5202 leader_election.cc:290] T c0837da4be774518841acb6daa85207d P ecdb77fe301541a4a0f652e78308dc80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f50d561a4167470b961a3f7dbeb6ef3e (127.2.197.194:37203), 12d3554b22244952a8cb77fa76ad9e38 (127.2.197.195:40127)
I20260429 14:03:49.059741 5216 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:49.057370 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:37203 (local address 127.2.197.193:43003)
0429 14:03:49.057661 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:49.057669 (+ 8us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:49.057736 (+ 67us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:49.057899 (+ 163us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:49.057900 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:49.057901 (+ 1us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:03:49.058109 (+ 208us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:49.058111 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:49.058851 (+ 740us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:49.058852 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:49.059482 (+ 630us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:49.059484 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:49.059582 (+ 98us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:49.059587 (+ 5us) client_negotiation.cc:770] Sending connection context
0429 14:03:49.059620 (+ 33us) client_negotiation.cc:241] Negotiation successful
0429 14:03:49.059654 (+ 34us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":237,"threads_started":1}
I20260429 14:03:49.059845 5214 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:49.057192 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:40127 (local address 127.2.197.193:38715)
0429 14:03:49.057445 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:49.057461 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:49.057579 (+ 118us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:49.057812 (+ 233us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:49.057813 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:49.057815 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:03:49.058329 (+ 514us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:49.058335 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:49.059003 (+ 668us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:49.059004 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:49.059605 (+ 601us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:49.059607 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:49.059694 (+ 87us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:49.059698 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 14:03:49.059730 (+ 32us) client_negotiation.cc:241] Negotiation successful
0429 14:03:49.059768 (+ 38us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":73,"threads_started":1}
I20260429 14:03:49.060249 5207 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:49.057421 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:43003 (local address 127.2.197.194:37203)
0429 14:03:49.057464 (+ 43us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:49.057466 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:49.057736 (+ 270us) server_negotiation.cc:408] Connection header received
0429 14:03:49.057796 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:49.057797 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:49.057809 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:49.057881 (+ 72us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:03:49.058189 (+ 308us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:49.058762 (+ 573us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:49.059579 (+ 817us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:49.060151 (+ 572us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:49.060159 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:49.060179 (+ 20us) server_negotiation.cc:300] Negotiation successful
0429 14:03:49.060205 (+ 26us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260429 14:03:49.060375 5215 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:49.057284 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:38715 (local address 127.2.197.195:40127)
0429 14:03:49.057599 (+ 315us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:49.057602 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:49.057632 (+ 30us) server_negotiation.cc:408] Connection header received
0429 14:03:49.057665 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:49.057666 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:49.057680 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:49.057800 (+ 120us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:03:49.058423 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:49.058916 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:49.059690 (+ 774us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:49.060222 (+ 532us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:49.060229 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:49.060249 (+ 20us) server_negotiation.cc:300] Negotiation successful
0429 14:03:49.060270 (+ 21us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":102,"threads_started":1}
I20260429 14:03:49.060485 4900 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c0837da4be774518841acb6daa85207d" candidate_uuid: "ecdb77fe301541a4a0f652e78308dc80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" is_pre_election: true
I20260429 14:03:49.060595 4900 raft_consensus.cc:2468] T c0837da4be774518841acb6daa85207d P f50d561a4167470b961a3f7dbeb6ef3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecdb77fe301541a4a0f652e78308dc80 in term 0.
I20260429 14:03:49.060644 5042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c0837da4be774518841acb6daa85207d" candidate_uuid: "ecdb77fe301541a4a0f652e78308dc80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d3554b22244952a8cb77fa76ad9e38" is_pre_election: true
I20260429 14:03:49.060779 5042 raft_consensus.cc:2468] T c0837da4be774518841acb6daa85207d P 12d3554b22244952a8cb77fa76ad9e38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecdb77fe301541a4a0f652e78308dc80 in term 0.
I20260429 14:03:49.060814 4690 leader_election.cc:304] T c0837da4be774518841acb6daa85207d P ecdb77fe301541a4a0f652e78308dc80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ecdb77fe301541a4a0f652e78308dc80, f50d561a4167470b961a3f7dbeb6ef3e; no voters:
I20260429 14:03:49.060941 5202 raft_consensus.cc:2804] T c0837da4be774518841acb6daa85207d P ecdb77fe301541a4a0f652e78308dc80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 14:03:49.060984 5202 raft_consensus.cc:493] T c0837da4be774518841acb6daa85207d P ecdb77fe301541a4a0f652e78308dc80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 14:03:49.061007 5202 raft_consensus.cc:3060] T c0837da4be774518841acb6daa85207d P ecdb77fe301541a4a0f652e78308dc80 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:03:49.061590 5202 raft_consensus.cc:515] T c0837da4be774518841acb6daa85207d P ecdb77fe301541a4a0f652e78308dc80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:49.061728 5202 leader_election.cc:290] T c0837da4be774518841acb6daa85207d P ecdb77fe301541a4a0f652e78308dc80 [CANDIDATE]: Term 1 election: Requested vote from peers f50d561a4167470b961a3f7dbeb6ef3e (127.2.197.194:37203), 12d3554b22244952a8cb77fa76ad9e38 (127.2.197.195:40127)
I20260429 14:03:49.061939 5042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c0837da4be774518841acb6daa85207d" candidate_uuid: "ecdb77fe301541a4a0f652e78308dc80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d3554b22244952a8cb77fa76ad9e38"
I20260429 14:03:49.061955 4900 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c0837da4be774518841acb6daa85207d" candidate_uuid: "ecdb77fe301541a4a0f652e78308dc80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f50d561a4167470b961a3f7dbeb6ef3e"
I20260429 14:03:49.062019 5042 raft_consensus.cc:3060] T c0837da4be774518841acb6daa85207d P 12d3554b22244952a8cb77fa76ad9e38 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:03:49.062019 4900 raft_consensus.cc:3060] T c0837da4be774518841acb6daa85207d P f50d561a4167470b961a3f7dbeb6ef3e [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:03:49.062785 5042 raft_consensus.cc:2468] T c0837da4be774518841acb6daa85207d P 12d3554b22244952a8cb77fa76ad9e38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecdb77fe301541a4a0f652e78308dc80 in term 1.
I20260429 14:03:49.062785 4900 raft_consensus.cc:2468] T c0837da4be774518841acb6daa85207d P f50d561a4167470b961a3f7dbeb6ef3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecdb77fe301541a4a0f652e78308dc80 in term 1.
I20260429 14:03:49.063002 4692 leader_election.cc:304] T c0837da4be774518841acb6daa85207d P ecdb77fe301541a4a0f652e78308dc80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12d3554b22244952a8cb77fa76ad9e38, ecdb77fe301541a4a0f652e78308dc80; no voters:
I20260429 14:03:49.063194 5202 raft_consensus.cc:2804] T c0837da4be774518841acb6daa85207d P ecdb77fe301541a4a0f652e78308dc80 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:03:49.063321 5202 raft_consensus.cc:697] T c0837da4be774518841acb6daa85207d P ecdb77fe301541a4a0f652e78308dc80 [term 1 LEADER]: Becoming Leader. State: Replica: ecdb77fe301541a4a0f652e78308dc80, State: Running, Role: LEADER
I20260429 14:03:49.063377 5202 consensus_queue.cc:237] T c0837da4be774518841acb6daa85207d P ecdb77fe301541a4a0f652e78308dc80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } }
I20260429 14:03:49.064155 4599 catalog_manager.cc:5671] T c0837da4be774518841acb6daa85207d P ecdb77fe301541a4a0f652e78308dc80 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecdb77fe301541a4a0f652e78308dc80 (127.2.197.193). New cstate: current_term: 1 leader_uuid: "ecdb77fe301541a4a0f652e78308dc80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdb77fe301541a4a0f652e78308dc80" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 } health_report { overall_health: UNKNOWN } } }
I20260429 14:03:49.111053 5219 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:49.108989 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:50062 (local address 127.2.197.193:42969)
0429 14:03:49.109144 (+ 155us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:49.109146 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:49.109154 (+ 8us) server_negotiation.cc:408] Connection header received
0429 14:03:49.109187 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:49.109188 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:49.109202 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:49.109255 (+ 53us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:03:49.109496 (+ 241us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:49.110025 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:49.110536 (+ 511us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:49.110653 (+ 117us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:49.110677 (+ 24us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:03:49.110759 (+ 82us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:03:49.110813 (+ 54us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:49.110876 (+ 63us) server_negotiation.cc:300] Negotiation successful
0429 14:03:49.110894 (+ 18us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":93,"threads_started":1}
I20260429 14:03:49.115422 5222 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:49.113341 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:40127 (local address 127.2.197.193:36253)
0429 14:03:49.113565 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:49.113576 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:49.113634 (+ 58us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:49.113753 (+ 119us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:49.113754 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:49.113756 (+ 2us) client_negotiation.cc:190] Negotiated authn=TOKEN
0429 14:03:49.113924 (+ 168us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:49.113925 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:49.114605 (+ 680us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:49.114606 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:49.115015 (+ 409us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:49.115017 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:49.115079 (+ 62us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:49.115090 (+ 11us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0429 14:03:49.115287 (+ 197us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0429 14:03:49.115289 (+ 2us) client_negotiation.cc:770] Sending connection context
0429 14:03:49.115326 (+ 37us) client_negotiation.cc:241] Negotiation successful
0429 14:03:49.115359 (+ 33us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":86,"threads_started":1}
I20260429 14:03:49.115422 5215 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:49.113424 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:36253 (local address 127.2.197.195:40127)
0429 14:03:49.113525 (+ 101us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:49.113527 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:49.113635 (+ 108us) server_negotiation.cc:408] Connection header received
0429 14:03:49.113674 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:49.113675 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:49.113684 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:49.113731 (+ 47us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:03:49.113981 (+ 250us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:49.114496 (+ 515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:49.115084 (+ 588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:49.115190 (+ 106us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:49.115205 (+ 15us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:03:49.115226 (+ 21us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:03:49.115268 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:49.115352 (+ 84us) server_negotiation.cc:300] Negotiation successful
0429 14:03:49.115370 (+ 18us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260429 14:03:49.117619 5042 raft_consensus.cc:1275] T c0837da4be774518841acb6daa85207d P 12d3554b22244952a8cb77fa76ad9e38 [term 1 FOLLOWER]: Refusing update from remote peer ecdb77fe301541a4a0f652e78308dc80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:03:49.117643 4900 raft_consensus.cc:1275] T c0837da4be774518841acb6daa85207d P f50d561a4167470b961a3f7dbeb6ef3e [term 1 FOLLOWER]: Refusing update from remote peer ecdb77fe301541a4a0f652e78308dc80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:03:49.117877 5217 consensus_queue.cc:1048] T c0837da4be774518841acb6daa85207d P ecdb77fe301541a4a0f652e78308dc80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12d3554b22244952a8cb77fa76ad9e38" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:03:49.118005 5202 consensus_queue.cc:1048] T c0837da4be774518841acb6daa85207d P ecdb77fe301541a4a0f652e78308dc80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f50d561a4167470b961a3f7dbeb6ef3e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:03:49.125946 5227 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:49.123065 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:42969 (local address 127.2.197.195:41725)
0429 14:03:49.123198 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:49.123209 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:49.123275 (+ 66us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:49.123463 (+ 188us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:49.123464 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:49.123466 (+ 2us) client_negotiation.cc:190] Negotiated authn=TOKEN
0429 14:03:49.123737 (+ 271us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:49.123739 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:49.124606 (+ 867us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:49.124607 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:49.125222 (+ 615us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:49.125224 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:49.125559 (+ 335us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:49.125565 (+ 6us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0429 14:03:49.125734 (+ 169us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0429 14:03:49.125736 (+ 2us) client_negotiation.cc:770] Sending connection context
0429 14:03:49.125813 (+ 77us) client_negotiation.cc:241] Negotiation successful
0429 14:03:49.125864 (+ 51us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":88,"threads_started":1}
I20260429 14:03:49.125938 5219 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:49.123096 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:41725 (local address 127.2.197.193:42969)
0429 14:03:49.123143 (+ 47us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:49.123145 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:49.123357 (+ 212us) server_negotiation.cc:408] Connection header received
0429 14:03:49.123385 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:49.123387 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:49.123400 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:49.123460 (+ 60us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:03:49.123988 (+ 528us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:49.124476 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:49.125327 (+ 851us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:49.125530 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:49.125641 (+ 111us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:03:49.125679 (+ 38us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:03:49.125734 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:49.125842 (+ 108us) server_negotiation.cc:300] Negotiation successful
0429 14:03:49.125870 (+ 28us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260429 14:03:49.126724 5170 tablet_replica.cc:1307] Aborted: operation has been aborted: cancelling pending write operations
I20260429 14:03:50.659672 4599 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:03:50.659789 4599 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:03:50.660848 4599 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 1aeac3f977874eb79cf735db9587a3eb: Sending DeleteTablet for 3 replicas of tablet c0837da4be774518841acb6daa85207d
I20260429 14:03:50.661340 4738 tablet_service.cc:1558] Processing DeleteTablet for tablet c0837da4be774518841acb6daa85207d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:03:50 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:52900
I20260429 14:03:50.661420 4880 tablet_service.cc:1558] Processing DeleteTablet for tablet c0837da4be774518841acb6daa85207d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:03:50 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:45904
I20260429 14:03:50.661566 5021 tablet_service.cc:1558] Processing DeleteTablet for tablet c0837da4be774518841acb6daa85207d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:03:50 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:49430
I20260429 14:03:50.661600 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 4673
I20260429 14:03:50.662518 5252 tablet_replica.cc:333] T c0837da4be774518841acb6daa85207d P 12d3554b22244952a8cb77fa76ad9e38: stopping tablet replica
I20260429 14:03:50.662603 5252 raft_consensus.cc:2243] T c0837da4be774518841acb6daa85207d P 12d3554b22244952a8cb77fa76ad9e38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:03:50.662658 5252 raft_consensus.cc:2272] T c0837da4be774518841acb6daa85207d P 12d3554b22244952a8cb77fa76ad9e38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:03:50.662971 5252 ts_tablet_manager.cc:1916] T c0837da4be774518841acb6daa85207d P 12d3554b22244952a8cb77fa76ad9e38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:03:50.664726 5252 ts_tablet_manager.cc:1929] T c0837da4be774518841acb6daa85207d P 12d3554b22244952a8cb77fa76ad9e38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260429 14:03:50.664795 5252 log.cc:1199] T c0837da4be774518841acb6daa85207d P 12d3554b22244952a8cb77fa76ad9e38: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777471413505459-2839-0/minicluster-data/ts-2/wal/wals/c0837da4be774518841acb6daa85207d
I20260429 14:03:50.665131 5252 ts_tablet_manager.cc:1950] T c0837da4be774518841acb6daa85207d P 12d3554b22244952a8cb77fa76ad9e38: Deleting consensus metadata
I20260429 14:03:50.665539 4586 catalog_manager.cc:5002] TS 12d3554b22244952a8cb77fa76ad9e38 (127.2.197.195:40127): tablet c0837da4be774518841acb6daa85207d (table test-table [id=d6f692f065b54d4aaf68bff9f6d13f36]) successfully deleted
W20260429 14:03:50.670006 4584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.197.193:42969 (error 108)
I20260429 14:03:50.670141 5251 tablet_replica.cc:333] T c0837da4be774518841acb6daa85207d P f50d561a4167470b961a3f7dbeb6ef3e: stopping tablet replica
I20260429 14:03:50.670212 5251 raft_consensus.cc:2243] T c0837da4be774518841acb6daa85207d P f50d561a4167470b961a3f7dbeb6ef3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:03:50.670260 5251 raft_consensus.cc:2272] T c0837da4be774518841acb6daa85207d P f50d561a4167470b961a3f7dbeb6ef3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:03:50.670414 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 4815
I20260429 14:03:50.670697 5251 ts_tablet_manager.cc:1916] T c0837da4be774518841acb6daa85207d P f50d561a4167470b961a3f7dbeb6ef3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:03:50.672425 5254 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:50.670612 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:42969 (local address 127.0.0.1:50070)
0429 14:03:50.672323 (+ 1711us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:50.672374 (+ 51us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.193:42969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1674,"thread_start_us":86,"threads_started":1}
W20260429 14:03:50.672771 4584 catalog_manager.cc:4729] TS ecdb77fe301541a4a0f652e78308dc80 (127.2.197.193:42969): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c0837da4be774518841acb6daa85207d: Network error: Client connection negotiation failed: client connection to 127.2.197.193:42969: connect: Connection refused (error 111)
I20260429 14:03:50.676774 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 4957
I20260429 14:03:50.676819 5254 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:50.676659 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:37203 (local address 127.0.0.1:54434)
0429 14:03:50.676750 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:50.676781 (+ 31us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.194:37203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260429 14:03:50.683383 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 4567
2026-04-29T14:03:50Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7160 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:03:50 dist-test-slave-8n77 krb5kdc[5257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:03:50 dist-test-slave-8n77 krb5kdc[5257](info): set up 2 sockets
Apr 29 14:03:50 dist-test-slave-8n77 krb5kdc[5257](info): commencing operation
krb5kdc: starting...
W20260429 14:03:52.786229 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.079s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260429 14:03:53.102416 2839 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.316s user 0.002s sys 0.000s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260429 14:03:53.394488 2839 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.292s user 0.002s sys 0.000s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260429 14:03:53.693125 2839 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.298s user 0.001s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/krb5kdc/test-user.keytab.
W20260429 14:03:54.839880 2839 mini_kdc.cc:290] Time spent creating keytab for test-user: real 1.147s user 0.000s sys 0.002s
Apr 29 14:03:54 dist-test-slave-8n77 krb5kdc[5257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:03:54Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:03:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab.
W20260429 14:03:55.933943 2839 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.2.197.254: real 1.072s user 0.000s sys 0.003s
WARNING: no policy specified for HTTP/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:03:55.963730 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:37777
--webserver_interface=127.2.197.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:43375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:37777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:03:56.043983 5273 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:03:56.044157 5273 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:03:56.044179 5273 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:03:56.045696 5273 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:03:56.045746 5273 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:03:56.045761 5273 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:03:56.045773 5273 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:03:56.045785 5273 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:03:56.047501 5273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:43375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:37777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:37777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:03:56.047750 5273 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:03:56.047960 5273 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:03:56.050434 5278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:03:56.050434 5281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:03:56.050539 5273 server_base.cc:1061] running on GCE node
W20260429 14:03:56.050515 5279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:03:56.050920 5273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:03:56.051138 5273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:03:56.052277 5273 hybrid_clock.cc:648] HybridClock initialized: now 1777471436052255 us; error 41 us; skew 500 ppm
Apr 29 14:03:56 dist-test-slave-8n77 krb5kdc[5257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471436, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:03:56.055917 5273 init.cc:377] Logged in from keytab as kudu/127.2.197.254@KRBTEST.COM (short username kudu)
I20260429 14:03:56.056649 5273 webserver.cc:492] Webserver started at http://127.2.197.254:34573/ using document root <none> and password file <none>
I20260429 14:03:56.056876 5273 fs_manager.cc:362] Metadata directory not provided
I20260429 14:03:56.056936 5273 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:03:56.057042 5273 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:03:56.058413 5273 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "504e0fb989ec4161874db19baf15c2c7"
format_stamp: "Formatted at 2026-04-29 14:03:56 on dist-test-slave-8n77"
server_key: "44a8035937ba3d1a34d8d6a491ffb0bd"
server_key_iv: "63bf9fdf41e1e9e006dce6aaa1e3fa7c"
server_key_version: "encryptionkey@0"
I20260429 14:03:56.058816 5273 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "504e0fb989ec4161874db19baf15c2c7"
format_stamp: "Formatted at 2026-04-29 14:03:56 on dist-test-slave-8n77"
server_key: "44a8035937ba3d1a34d8d6a491ffb0bd"
server_key_iv: "63bf9fdf41e1e9e006dce6aaa1e3fa7c"
server_key_version: "encryptionkey@0"
I20260429 14:03:56.060449 5273 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:03:56.061348 5288 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:56.061641 5273 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:03:56.061721 5273 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "504e0fb989ec4161874db19baf15c2c7"
format_stamp: "Formatted at 2026-04-29 14:03:56 on dist-test-slave-8n77"
server_key: "44a8035937ba3d1a34d8d6a491ffb0bd"
server_key_iv: "63bf9fdf41e1e9e006dce6aaa1e3fa7c"
server_key_version: "encryptionkey@0"
I20260429 14:03:56.061796 5273 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:03:56.075459 5273 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:03:56.080772 5273 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:03:56.080894 5273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:03:56.084919 5273 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:37777
I20260429 14:03:56.085023 5340 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:37777 every 8 connection(s)
I20260429 14:03:56.085289 5273 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:03:56.085865 5341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:03:56.088199 5341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 504e0fb989ec4161874db19baf15c2c7: Bootstrap starting.
I20260429 14:03:56.088762 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 5273
I20260429 14:03:56.088837 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:03:56.088806 5341 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 504e0fb989ec4161874db19baf15c2c7: Neither blocks nor log segments found. Creating new log.
I20260429 14:03:56.089010 2839 external_mini_cluster.cc:1468] Setting key 6e8229731d9017301ef2fc8ebbd59a97
I20260429 14:03:56.089083 5341 log.cc:826] T 00000000000000000000000000000000 P 504e0fb989ec4161874db19baf15c2c7: Log is configured to *not* fsync() on all Append() calls
I20260429 14:03:56.089890 5341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 504e0fb989ec4161874db19baf15c2c7: No bootstrap required, opened a new log
I20260429 14:03:56.091447 5341 raft_consensus.cc:359] T 00000000000000000000000000000000 P 504e0fb989ec4161874db19baf15c2c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "504e0fb989ec4161874db19baf15c2c7" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 37777 } }
I20260429 14:03:56.091560 5341 raft_consensus.cc:385] T 00000000000000000000000000000000 P 504e0fb989ec4161874db19baf15c2c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:03:56.091584 5341 raft_consensus.cc:740] T 00000000000000000000000000000000 P 504e0fb989ec4161874db19baf15c2c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 504e0fb989ec4161874db19baf15c2c7, State: Initialized, Role: FOLLOWER
I20260429 14:03:56.091681 5341 consensus_queue.cc:260] T 00000000000000000000000000000000 P 504e0fb989ec4161874db19baf15c2c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "504e0fb989ec4161874db19baf15c2c7" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 37777 } }
I20260429 14:03:56.091743 5341 raft_consensus.cc:399] T 00000000000000000000000000000000 P 504e0fb989ec4161874db19baf15c2c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:03:56.091768 5341 raft_consensus.cc:493] T 00000000000000000000000000000000 P 504e0fb989ec4161874db19baf15c2c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:03:56.091810 5341 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 504e0fb989ec4161874db19baf15c2c7 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:03:56.092509 5341 raft_consensus.cc:515] T 00000000000000000000000000000000 P 504e0fb989ec4161874db19baf15c2c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "504e0fb989ec4161874db19baf15c2c7" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 37777 } }
I20260429 14:03:56.092619 5341 leader_election.cc:304] T 00000000000000000000000000000000 P 504e0fb989ec4161874db19baf15c2c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 504e0fb989ec4161874db19baf15c2c7; no voters:
I20260429 14:03:56.092836 5341 leader_election.cc:290] T 00000000000000000000000000000000 P 504e0fb989ec4161874db19baf15c2c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:03:56.092867 5346 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 504e0fb989ec4161874db19baf15c2c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:03:56.093075 5346 raft_consensus.cc:697] T 00000000000000000000000000000000 P 504e0fb989ec4161874db19baf15c2c7 [term 1 LEADER]: Becoming Leader. State: Replica: 504e0fb989ec4161874db19baf15c2c7, State: Running, Role: LEADER
I20260429 14:03:56.093149 5341 sys_catalog.cc:565] T 00000000000000000000000000000000 P 504e0fb989ec4161874db19baf15c2c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:03:56.093220 5346 consensus_queue.cc:237] T 00000000000000000000000000000000 P 504e0fb989ec4161874db19baf15c2c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "504e0fb989ec4161874db19baf15c2c7" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 37777 } }
I20260429 14:03:56.093547 5346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 504e0fb989ec4161874db19baf15c2c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 504e0fb989ec4161874db19baf15c2c7. Latest consensus state: current_term: 1 leader_uuid: "504e0fb989ec4161874db19baf15c2c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "504e0fb989ec4161874db19baf15c2c7" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 37777 } } }
I20260429 14:03:56.093559 5348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 504e0fb989ec4161874db19baf15c2c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "504e0fb989ec4161874db19baf15c2c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "504e0fb989ec4161874db19baf15c2c7" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 37777 } } }
I20260429 14:03:56.093859 5348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 504e0fb989ec4161874db19baf15c2c7 [sys.catalog]: This master's current role is: LEADER
I20260429 14:03:56.093739 5346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 504e0fb989ec4161874db19baf15c2c7 [sys.catalog]: This master's current role is: LEADER
Apr 29 14:03:56 dist-test-slave-8n77 krb5kdc[5257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
W20260429 14:03:56.096092 5362 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 504e0fb989ec4161874db19baf15c2c7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260429 14:03:56.096149 5362 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260429 14:03:56.096199 5355 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:03:56.096366 5355 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:03:56.097878 5355 catalog_manager.cc:1357] Generated new cluster ID: 7ff2f45f18504270a4a169472edc4346
I20260429 14:03:56.097923 5355 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:03:56.099802 5345 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:56.089556 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51826 (local address 127.2.197.254:37777)
0429 14:03:56.089733 (+ 177us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:56.089744 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:56.089766 (+ 22us) server_negotiation.cc:408] Connection header received
0429 14:03:56.090129 (+ 363us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:56.090136 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:56.090175 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:56.090330 (+ 155us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:56.091242 (+ 912us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:56.092173 (+ 931us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:56.092843 (+ 670us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:56.093139 (+ 296us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:56.096281 (+ 3142us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:56.096290 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:56.096291 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:56.096304 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:56.098408 (+ 2104us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:56.098800 (+ 392us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:56.098803 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:56.098803 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:56.098854 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:56.099021 (+ 167us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:56.099022 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:56.099022 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:56.099227 (+ 205us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:56.099373 (+ 146us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:56.099518 (+ 145us) server_negotiation.cc:300] Negotiation successful
0429 14:03:56.099641 (+ 123us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":109,"threads_started":1}
I20260429 14:03:56.116024 5355 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:03:56.116782 5355 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:03:56.121999 5355 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 504e0fb989ec4161874db19baf15c2c7: Generated new TSK 0
I20260429 14:03:56.122187 5355 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:03:56.182662 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.193:0
--local_ip_for_outbound_sockets=127.2.197.193
--webserver_interface=127.2.197.193
--webserver_port=0
--tserver_master_addrs=127.2.197.254:37777
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:43375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:03:56.266975 5369 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:03:56.267158 5369 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:03:56.267184 5369 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:03:56.268724 5369 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:03:56.268771 5369 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:03:56.268800 5369 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.193
I20260429 14:03:56.270524 5369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:43375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.197.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:37777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.193
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:03:56.270767 5369 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:03:56.270958 5369 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:03:56.273638 5374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:03:56.273757 5369 server_base.cc:1061] running on GCE node
W20260429 14:03:56.273638 5375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:03:56.273648 5377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:03:56.274086 5369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:03:56.274313 5369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:03:56.275482 5369 hybrid_clock.cc:648] HybridClock initialized: now 1777471436275465 us; error 33 us; skew 500 ppm
Apr 29 14:03:56 dist-test-slave-8n77 krb5kdc[5257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471436, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:03:56.277961 5369 init.cc:377] Logged in from keytab as kudu/127.2.197.193@KRBTEST.COM (short username kudu)
I20260429 14:03:56.278580 5369 webserver.cc:492] Webserver started at http://127.2.197.193:42809/ using document root <none> and password file <none>
I20260429 14:03:56.278774 5369 fs_manager.cc:362] Metadata directory not provided
I20260429 14:03:56.278836 5369 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:03:56.278939 5369 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:03:56.279886 5369 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-0/data/instance:
uuid: "ad584fefbdb7453a94bed7e8c0d2ec61"
format_stamp: "Formatted at 2026-04-29 14:03:56 on dist-test-slave-8n77"
server_key: "5e22592515473c9f4e863dfed21690d1"
server_key_iv: "1906629c61c234656d8cbdb6052dcb62"
server_key_version: "encryptionkey@0"
I20260429 14:03:56.280211 5369 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance:
uuid: "ad584fefbdb7453a94bed7e8c0d2ec61"
format_stamp: "Formatted at 2026-04-29 14:03:56 on dist-test-slave-8n77"
server_key: "5e22592515473c9f4e863dfed21690d1"
server_key_iv: "1906629c61c234656d8cbdb6052dcb62"
server_key_version: "encryptionkey@0"
I20260429 14:03:56.281404 5369 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260429 14:03:56.282173 5384 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:56.282500 5369 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260429 14:03:56.282634 5369 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-0/wal
uuid: "ad584fefbdb7453a94bed7e8c0d2ec61"
format_stamp: "Formatted at 2026-04-29 14:03:56 on dist-test-slave-8n77"
server_key: "5e22592515473c9f4e863dfed21690d1"
server_key_iv: "1906629c61c234656d8cbdb6052dcb62"
server_key_version: "encryptionkey@0"
I20260429 14:03:56.282716 5369 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:03:56.296890 5369 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:03:56.298935 5369 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:03:56.299100 5369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:03:56.299328 5369 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:03:56.299650 5369 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:03:56.299702 5369 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:56.299742 5369 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:03:56.299800 5369 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:56.305521 5369 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.193:43303
I20260429 14:03:56.305629 5497 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.193:43303 every 8 connection(s)
I20260429 14:03:56.305899 5369 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
I20260429 14:03:56.307425 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 5369
I20260429 14:03:56.307519 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance
I20260429 14:03:56.307680 2839 external_mini_cluster.cc:1468] Setting key 7408730f3f6d16b564ac17d4f83cbafb
Apr 29 14:03:56 dist-test-slave-8n77 krb5kdc[5257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471436, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.197.194@KRBTEST.COM; defaulting to no policy
I20260429 14:03:56.316177 5345 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:56.306794 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:54869 (local address 127.2.197.254:37777)
0429 14:03:56.306865 (+ 71us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:56.306867 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:56.307084 (+ 217us) server_negotiation.cc:408] Connection header received
0429 14:03:56.307826 (+ 742us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:56.307828 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:56.307840 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:56.307919 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:56.308867 (+ 948us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:56.309642 (+ 775us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:56.310464 (+ 822us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:56.310619 (+ 155us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:56.313349 (+ 2730us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:56.313353 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:56.313353 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:56.313364 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:56.314888 (+ 1524us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:56.315308 (+ 420us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:56.315309 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:56.315309 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:56.315340 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:56.315541 (+ 201us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:56.315542 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:56.315543 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:56.315649 (+ 106us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:56.315724 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:56.315976 (+ 252us) server_negotiation.cc:300] Negotiation successful
0429 14:03:56.316079 (+ 103us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260429 14:03:56.316274 5500 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:56.306793 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37777 (local address 127.2.197.193:54869)
0429 14:03:56.306960 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:56.306994 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:56.307692 (+ 698us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:56.308008 (+ 316us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:56.308012 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:56.308451 (+ 439us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:03:56.308754 (+ 303us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:56.308758 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:56.309747 (+ 989us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:56.309749 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:56.310370 (+ 621us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:56.310373 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:56.310526 (+ 153us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:56.311624 (+ 1098us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:03:56.311637 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:03:56.313250 (+ 1613us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:03:56.315005 (+ 1755us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:56.315011 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:56.315014 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:56.315219 (+ 205us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:56.315403 (+ 184us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:56.315404 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:56.315405 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:56.315469 (+ 64us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:56.315714 (+ 245us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:03:56.315719 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:03:56.315845 (+ 126us) client_negotiation.cc:770] Sending connection context
0429 14:03:56.315956 (+ 111us) client_negotiation.cc:241] Negotiation successful
0429 14:03:56.316091 (+ 135us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":100,"threads_started":1}
I20260429 14:03:56.316716 5498 heartbeater.cc:344] Connected to a master server at 127.2.197.254:37777
I20260429 14:03:56.316810 5498 heartbeater.cc:461] Registering TS with master...
I20260429 14:03:56.317008 5498 heartbeater.cc:507] Master 127.2.197.254:37777 requested a full tablet report, sending...
I20260429 14:03:56.317550 5305 ts_manager.cc:194] Registered new tserver with Master: ad584fefbdb7453a94bed7e8c0d2ec61 (127.2.197.193:43303)
I20260429 14:03:56.318264 5305 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.193@KRBTEST.COM'} at 127.2.197.193:54869
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:03:56.362654 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.194:0
--local_ip_for_outbound_sockets=127.2.197.194
--webserver_interface=127.2.197.194
--webserver_port=0
--tserver_master_addrs=127.2.197.254:37777
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:43375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:03:56.441186 5505 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:03:56.441372 5505 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:03:56.441397 5505 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:03:56.442952 5505 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:03:56.443001 5505 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:03:56.443028 5505 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.194
I20260429 14:03:56.444649 5505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:43375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.197.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:37777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.194
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:03:56.444867 5505 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:03:56.445053 5505 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:03:56.447584 5511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:03:56.447482 5513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:03:56.447665 5510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:03:56.447916 5505 server_base.cc:1061] running on GCE node
I20260429 14:03:56.448103 5505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:03:56.448268 5505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:03:56.449412 5505 hybrid_clock.cc:648] HybridClock initialized: now 1777471436449381 us; error 55 us; skew 500 ppm
Apr 29 14:03:56 dist-test-slave-8n77 krb5kdc[5257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471436, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:03:56.451992 5505 init.cc:377] Logged in from keytab as kudu/127.2.197.194@KRBTEST.COM (short username kudu)
I20260429 14:03:56.452502 5505 webserver.cc:492] Webserver started at http://127.2.197.194:33303/ using document root <none> and password file <none>
I20260429 14:03:56.452694 5505 fs_manager.cc:362] Metadata directory not provided
I20260429 14:03:56.452735 5505 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:03:56.452840 5505 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:03:56.453795 5505 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-1/data/instance:
uuid: "14e511715ee641e2b496ae49a51a8341"
format_stamp: "Formatted at 2026-04-29 14:03:56 on dist-test-slave-8n77"
server_key: "3f54b614db8eb64f7267314f7052c1ed"
server_key_iv: "1d1602bf3609672a532b35dec485aa23"
server_key_version: "encryptionkey@0"
I20260429 14:03:56.454118 5505 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance:
uuid: "14e511715ee641e2b496ae49a51a8341"
format_stamp: "Formatted at 2026-04-29 14:03:56 on dist-test-slave-8n77"
server_key: "3f54b614db8eb64f7267314f7052c1ed"
server_key_iv: "1d1602bf3609672a532b35dec485aa23"
server_key_version: "encryptionkey@0"
I20260429 14:03:56.455367 5505 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:03:56.456158 5520 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:56.456336 5505 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260429 14:03:56.456416 5505 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-1/wal
uuid: "14e511715ee641e2b496ae49a51a8341"
format_stamp: "Formatted at 2026-04-29 14:03:56 on dist-test-slave-8n77"
server_key: "3f54b614db8eb64f7267314f7052c1ed"
server_key_iv: "1d1602bf3609672a532b35dec485aa23"
server_key_version: "encryptionkey@0"
I20260429 14:03:56.456496 5505 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:03:56.475219 5505 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:03:56.476971 5505 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:03:56.477128 5505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:03:56.477363 5505 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:03:56.477720 5505 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:03:56.477777 5505 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:56.477808 5505 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:03:56.477829 5505 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:56.484112 5505 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.194:38917
I20260429 14:03:56.484151 5633 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.194:38917 every 8 connection(s)
I20260429 14:03:56.484442 5505 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
I20260429 14:03:56.487736 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 5505
I20260429 14:03:56.487850 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance
I20260429 14:03:56.488006 2839 external_mini_cluster.cc:1468] Setting key 157e9c3ef1a49c65584d1b655a78ebc7
Apr 29 14:03:56 dist-test-slave-8n77 krb5kdc[5257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471436, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:03:56.494529 5345 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:56.485155 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:33939 (local address 127.2.197.254:37777)
0429 14:03:56.485248 (+ 93us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:56.485251 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:56.485424 (+ 173us) server_negotiation.cc:408] Connection header received
0429 14:03:56.485912 (+ 488us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:56.485913 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:56.485924 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:56.485989 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:56.486892 (+ 903us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:56.487412 (+ 520us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:56.488312 (+ 900us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:56.488487 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:56.491320 (+ 2833us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:56.491326 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:56.491326 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:56.491335 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:56.493020 (+ 1685us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:56.493491 (+ 471us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:56.493492 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:56.493492 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:56.493526 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:56.493780 (+ 254us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:56.493781 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:56.493782 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:56.493946 (+ 164us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:56.494036 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:56.494330 (+ 294us) server_negotiation.cc:300] Negotiation successful
0429 14:03:56.494435 (+ 105us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260429 14:03:56.494650 5636 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:56.485119 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37777 (local address 127.2.197.194:33939)
0429 14:03:56.485312 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:56.485348 (+ 36us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:56.485806 (+ 458us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:56.486103 (+ 297us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:56.486107 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:56.486485 (+ 378us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:03:56.486779 (+ 294us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:56.486782 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:56.487492 (+ 710us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:56.487493 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:56.488118 (+ 625us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:56.488121 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:56.488323 (+ 202us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:56.489473 (+ 1150us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:03:56.489487 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:03:56.491214 (+ 1727us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:03:56.493155 (+ 1941us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:56.493162 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:56.493165 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:56.493394 (+ 229us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:56.493639 (+ 245us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:56.493640 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:56.493641 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:56.493712 (+ 71us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:56.494023 (+ 311us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:03:56.494028 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:03:56.494160 (+ 132us) client_negotiation.cc:770] Sending connection context
0429 14:03:56.494278 (+ 118us) client_negotiation.cc:241] Negotiation successful
0429 14:03:56.494462 (+ 184us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":82,"threads_started":1}
I20260429 14:03:56.495256 5634 heartbeater.cc:344] Connected to a master server at 127.2.197.254:37777
I20260429 14:03:56.495340 5634 heartbeater.cc:461] Registering TS with master...
I20260429 14:03:56.495548 5634 heartbeater.cc:507] Master 127.2.197.254:37777 requested a full tablet report, sending...
I20260429 14:03:56.495970 5305 ts_manager.cc:194] Registered new tserver with Master: 14e511715ee641e2b496ae49a51a8341 (127.2.197.194:38917)
I20260429 14:03:56.496532 5305 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.194@KRBTEST.COM'} at 127.2.197.194:33939
WARNING: no policy specified for kudu/127.2.197.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:03:56.542969 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.195:0
--local_ip_for_outbound_sockets=127.2.197.195
--webserver_interface=127.2.197.195
--webserver_port=0
--tserver_master_addrs=127.2.197.254:37777
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:43375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:03:56.620457 5641 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:03:56.620652 5641 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:03:56.620676 5641 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:03:56.622378 5641 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:03:56.622453 5641 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:03:56.622499 5641 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.195
I20260429 14:03:56.624411 5641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:43375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.197.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:37777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.195
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:03:56.624645 5641 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:03:56.624794 5641 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:03:56.627542 5647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:03:56.627579 5646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:03:56.627784 5649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:03:56.627969 5641 server_base.cc:1061] running on GCE node
I20260429 14:03:56.628182 5641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:03:56.628407 5641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:03:56.629582 5641 hybrid_clock.cc:648] HybridClock initialized: now 1777471436629559 us; error 50 us; skew 500 ppm
Apr 29 14:03:56 dist-test-slave-8n77 krb5kdc[5257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471436, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:03:56.632573 5641 init.cc:377] Logged in from keytab as kudu/127.2.197.195@KRBTEST.COM (short username kudu)
I20260429 14:03:56.633165 5641 webserver.cc:492] Webserver started at http://127.2.197.195:41079/ using document root <none> and password file <none>
I20260429 14:03:56.633361 5641 fs_manager.cc:362] Metadata directory not provided
I20260429 14:03:56.633419 5641 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:03:56.633528 5641 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:03:56.634572 5641 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-2/data/instance:
uuid: "242f40795f2940d091874f26b66fcafc"
format_stamp: "Formatted at 2026-04-29 14:03:56 on dist-test-slave-8n77"
server_key: "e2144d16673dfc70025855934318cda4"
server_key_iv: "9480505b83a714d6c5e182ec1057608c"
server_key_version: "encryptionkey@0"
I20260429 14:03:56.634917 5641 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance:
uuid: "242f40795f2940d091874f26b66fcafc"
format_stamp: "Formatted at 2026-04-29 14:03:56 on dist-test-slave-8n77"
server_key: "e2144d16673dfc70025855934318cda4"
server_key_iv: "9480505b83a714d6c5e182ec1057608c"
server_key_version: "encryptionkey@0"
I20260429 14:03:56.636195 5641 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260429 14:03:56.636991 5656 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:56.637208 5641 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:03:56.637279 5641 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-2/wal
uuid: "242f40795f2940d091874f26b66fcafc"
format_stamp: "Formatted at 2026-04-29 14:03:56 on dist-test-slave-8n77"
server_key: "e2144d16673dfc70025855934318cda4"
server_key_iv: "9480505b83a714d6c5e182ec1057608c"
server_key_version: "encryptionkey@0"
I20260429 14:03:56.637333 5641 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:03:56.647264 5641 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:03:56.649212 5641 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:03:56.649326 5641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:03:56.649593 5641 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:03:56.649953 5641 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:03:56.649988 5641 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:56.650012 5641 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:03:56.650061 5641 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:56.655984 5641 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.195:39555
I20260429 14:03:56.656050 5769 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.195:39555 every 8 connection(s)
I20260429 14:03:56.656352 5641 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
I20260429 14:03:56.657480 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 5641
I20260429 14:03:56.657634 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance
I20260429 14:03:56.657789 2839 external_mini_cluster.cc:1468] Setting key c83e673c4d17d65a28727fb96932e78e
Apr 29 14:03:56 dist-test-slave-8n77 krb5kdc[5257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471436, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:03:56.665693 5345 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:56.657059 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:48581 (local address 127.2.197.254:37777)
0429 14:03:56.657122 (+ 63us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:56.657124 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:56.657265 (+ 141us) server_negotiation.cc:408] Connection header received
0429 14:03:56.658012 (+ 747us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:56.658013 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:56.658028 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:56.658112 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:56.659237 (+ 1125us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:56.659732 (+ 495us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:56.660362 (+ 630us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:56.660542 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:56.663006 (+ 2464us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:56.663010 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:56.663010 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:56.663018 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:56.664388 (+ 1370us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:56.664854 (+ 466us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:56.664855 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:56.664856 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:56.664881 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:56.665078 (+ 197us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:56.665079 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:56.665080 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:56.665184 (+ 104us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:56.665262 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:56.665502 (+ 240us) server_negotiation.cc:300] Negotiation successful
0429 14:03:56.665596 (+ 94us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260429 14:03:56.665820 5772 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:56.657002 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37777 (local address 127.2.197.195:48581)
0429 14:03:56.657151 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:56.657183 (+ 32us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:56.657860 (+ 677us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:56.658177 (+ 317us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:56.658182 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:56.658656 (+ 474us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:03:56.659158 (+ 502us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:56.659163 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:56.659822 (+ 659us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:56.659823 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:56.660280 (+ 457us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:56.660282 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:56.660406 (+ 124us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:56.661313 (+ 907us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:03:56.661323 (+ 10us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:03:56.662908 (+ 1585us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:03:56.664532 (+ 1624us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:56.664536 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:56.664539 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:56.664771 (+ 232us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:56.664952 (+ 181us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:56.664954 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:56.664954 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:56.665017 (+ 63us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:56.665251 (+ 234us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:03:56.665255 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:03:56.665374 (+ 119us) client_negotiation.cc:770] Sending connection context
0429 14:03:56.665486 (+ 112us) client_negotiation.cc:241] Negotiation successful
0429 14:03:56.665629 (+ 143us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":91,"threads_started":1}
I20260429 14:03:56.666374 5770 heartbeater.cc:344] Connected to a master server at 127.2.197.254:37777
I20260429 14:03:56.666456 5770 heartbeater.cc:461] Registering TS with master...
I20260429 14:03:56.666671 5770 heartbeater.cc:507] Master 127.2.197.254:37777 requested a full tablet report, sending...
I20260429 14:03:56.667049 5305 ts_manager.cc:194] Registered new tserver with Master: 242f40795f2940d091874f26b66fcafc (127.2.197.195:39555)
I20260429 14:03:56.667438 5305 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.195@KRBTEST.COM'} at 127.2.197.195:48581
I20260429 14:03:56.670617 2839 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260429 14:03:56.681304 5345 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:03:56.680363 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51832 (local address 127.2.197.254:37777)
0429 14:03:56.680437 (+ 74us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:56.680439 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:56.680476 (+ 37us) server_negotiation.cc:408] Connection header received
0429 14:03:56.680551 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:56.680552 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:56.680564 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:56.680630 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:56.681204 (+ 574us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51832: BlockingRecv error: recv got EOF from 127.0.0.1:51832 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260429 14:03:56.682433 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 5369
I20260429 14:03:56.687001 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 5505
I20260429 14:03:56.691546 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 5641
I20260429 14:03:56.696434 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 5273
2026-04-29T14:03:56Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (6009 ms)
[ RUN ] SecurityITest.TestRebalanceReportUnauthorized
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:03:56 dist-test-slave-8n77 krb5kdc[5782](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:03:56 dist-test-slave-8n77 krb5kdc[5782](info): set up 2 sockets
Apr 29 14:03:56 dist-test-slave-8n77 krb5kdc[5782](info): commencing operation
krb5kdc: starting...
W20260429 14:03:58.744839 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:03:58 dist-test-slave-8n77 krb5kdc[5782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:03:58Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:03:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:03:58.889823 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:34587
--webserver_interface=127.2.197.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:39855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:34587
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:03:58.970593 5798 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:03:58.970852 5798 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:03:58.970896 5798 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:03:58.972532 5798 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:03:58.972635 5798 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:03:58.972658 5798 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:03:58.972680 5798 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:03:58.972711 5798 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:03:58.974615 5798 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:39855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:34587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:34587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:03:58.974910 5798 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:03:58.975109 5798 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:03:58.977900 5804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:03:58.977957 5798 server_base.cc:1061] running on GCE node
W20260429 14:03:58.977939 5806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:03:58.977905 5803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:03:58.978359 5798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:03:58.978631 5798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:03:58.979802 5798 hybrid_clock.cc:648] HybridClock initialized: now 1777471438979782 us; error 38 us; skew 500 ppm
Apr 29 14:03:58 dist-test-slave-8n77 krb5kdc[5782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471438, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:03:58.982584 5798 init.cc:377] Logged in from keytab as kudu/127.2.197.254@KRBTEST.COM (short username kudu)
I20260429 14:03:58.983130 5798 webserver.cc:492] Webserver started at http://127.2.197.254:46405/ using document root <none> and password file <none>
I20260429 14:03:58.983367 5798 fs_manager.cc:362] Metadata directory not provided
I20260429 14:03:58.983434 5798 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:03:58.983542 5798 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:03:58.984493 5798 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "df0fc0697e94470485d6496f41fa9021"
format_stamp: "Formatted at 2026-04-29 14:03:58 on dist-test-slave-8n77"
server_key: "b7bb10f3c4e2aeb424285bbbe493945b"
server_key_iv: "8b780d81db89ed371441be1049e84fde"
server_key_version: "encryptionkey@0"
I20260429 14:03:58.984822 5798 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "df0fc0697e94470485d6496f41fa9021"
format_stamp: "Formatted at 2026-04-29 14:03:58 on dist-test-slave-8n77"
server_key: "b7bb10f3c4e2aeb424285bbbe493945b"
server_key_iv: "8b780d81db89ed371441be1049e84fde"
server_key_version: "encryptionkey@0"
I20260429 14:03:58.986132 5798 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260429 14:03:58.986945 5813 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:58.987298 5798 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:03:58.987402 5798 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "df0fc0697e94470485d6496f41fa9021"
format_stamp: "Formatted at 2026-04-29 14:03:58 on dist-test-slave-8n77"
server_key: "b7bb10f3c4e2aeb424285bbbe493945b"
server_key_iv: "8b780d81db89ed371441be1049e84fde"
server_key_version: "encryptionkey@0"
I20260429 14:03:58.987484 5798 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:03:59.008963 5798 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:03:59.011016 5798 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:03:59.011193 5798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:03:59.015754 5798 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:34587
I20260429 14:03:59.015902 5865 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:34587 every 8 connection(s)
I20260429 14:03:59.016175 5798 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:03:59.016774 5866 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:03:59.019287 5866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df0fc0697e94470485d6496f41fa9021: Bootstrap starting.
I20260429 14:03:59.019968 5866 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df0fc0697e94470485d6496f41fa9021: Neither blocks nor log segments found. Creating new log.
I20260429 14:03:59.020251 5866 log.cc:826] T 00000000000000000000000000000000 P df0fc0697e94470485d6496f41fa9021: Log is configured to *not* fsync() on all Append() calls
I20260429 14:03:59.021075 5866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df0fc0697e94470485d6496f41fa9021: No bootstrap required, opened a new log
I20260429 14:03:59.022445 5866 raft_consensus.cc:359] T 00000000000000000000000000000000 P df0fc0697e94470485d6496f41fa9021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0fc0697e94470485d6496f41fa9021" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 34587 } }
I20260429 14:03:59.022576 5866 raft_consensus.cc:385] T 00000000000000000000000000000000 P df0fc0697e94470485d6496f41fa9021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:03:59.022601 5866 raft_consensus.cc:740] T 00000000000000000000000000000000 P df0fc0697e94470485d6496f41fa9021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df0fc0697e94470485d6496f41fa9021, State: Initialized, Role: FOLLOWER
I20260429 14:03:59.022782 5866 consensus_queue.cc:260] T 00000000000000000000000000000000 P df0fc0697e94470485d6496f41fa9021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0fc0697e94470485d6496f41fa9021" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 34587 } }
I20260429 14:03:59.022888 5866 raft_consensus.cc:399] T 00000000000000000000000000000000 P df0fc0697e94470485d6496f41fa9021 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:03:59.022929 5866 raft_consensus.cc:493] T 00000000000000000000000000000000 P df0fc0697e94470485d6496f41fa9021 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:03:59.023008 5866 raft_consensus.cc:3060] T 00000000000000000000000000000000 P df0fc0697e94470485d6496f41fa9021 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:03:59.023650 5866 raft_consensus.cc:515] T 00000000000000000000000000000000 P df0fc0697e94470485d6496f41fa9021 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0fc0697e94470485d6496f41fa9021" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 34587 } }
I20260429 14:03:59.023795 5866 leader_election.cc:304] T 00000000000000000000000000000000 P df0fc0697e94470485d6496f41fa9021 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df0fc0697e94470485d6496f41fa9021; no voters:
I20260429 14:03:59.023988 5866 leader_election.cc:290] T 00000000000000000000000000000000 P df0fc0697e94470485d6496f41fa9021 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:03:59.024080 5869 raft_consensus.cc:2804] T 00000000000000000000000000000000 P df0fc0697e94470485d6496f41fa9021 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:03:59.024260 5869 raft_consensus.cc:697] T 00000000000000000000000000000000 P df0fc0697e94470485d6496f41fa9021 [term 1 LEADER]: Becoming Leader. State: Replica: df0fc0697e94470485d6496f41fa9021, State: Running, Role: LEADER
I20260429 14:03:59.024257 5866 sys_catalog.cc:565] T 00000000000000000000000000000000 P df0fc0697e94470485d6496f41fa9021 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:03:59.024407 5869 consensus_queue.cc:237] T 00000000000000000000000000000000 P df0fc0697e94470485d6496f41fa9021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0fc0697e94470485d6496f41fa9021" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 34587 } }
I20260429 14:03:59.024722 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 5798
I20260429 14:03:59.024821 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:03:59.024784 5870 sys_catalog.cc:455] T 00000000000000000000000000000000 P df0fc0697e94470485d6496f41fa9021 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df0fc0697e94470485d6496f41fa9021" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0fc0697e94470485d6496f41fa9021" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 34587 } } }
I20260429 14:03:59.024833 5871 sys_catalog.cc:455] T 00000000000000000000000000000000 P df0fc0697e94470485d6496f41fa9021 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df0fc0697e94470485d6496f41fa9021. Latest consensus state: current_term: 1 leader_uuid: "df0fc0697e94470485d6496f41fa9021" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0fc0697e94470485d6496f41fa9021" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 34587 } } }
I20260429 14:03:59.024869 5870 sys_catalog.cc:458] T 00000000000000000000000000000000 P df0fc0697e94470485d6496f41fa9021 [sys.catalog]: This master's current role is: LEADER
I20260429 14:03:59.024896 5871 sys_catalog.cc:458] T 00000000000000000000000000000000 P df0fc0697e94470485d6496f41fa9021 [sys.catalog]: This master's current role is: LEADER
I20260429 14:03:59.024998 2839 external_mini_cluster.cc:1468] Setting key 9d913ad9eec8849e0e027191ceb9be71
I20260429 14:03:59.025583 5874 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:03:59.026292 5874 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:03:59.028023 5874 catalog_manager.cc:1357] Generated new cluster ID: b93c5faa749a4eb69d97f644905d13ee
I20260429 14:03:59.028077 5874 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Apr 29 14:03:59 dist-test-slave-8n77 krb5kdc[5782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:03:59.034502 5880 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:59.025491 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44346 (local address 127.2.197.254:34587)
0429 14:03:59.025850 (+ 359us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:59.025897 (+ 47us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:59.025920 (+ 23us) server_negotiation.cc:408] Connection header received
0429 14:03:59.026030 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:59.026036 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:59.026088 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:59.026247 (+ 159us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:59.027318 (+ 1071us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:59.028413 (+ 1095us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:59.029051 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:59.029310 (+ 259us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:59.031473 (+ 2163us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:59.031496 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:59.031497 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:59.031508 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:59.033140 (+ 1632us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:59.033498 (+ 358us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:59.033500 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:59.033501 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:59.033551 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:59.033712 (+ 161us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:59.033713 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:59.033713 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:59.033904 (+ 191us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:59.034062 (+ 158us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:59.034161 (+ 99us) server_negotiation.cc:300] Negotiation successful
0429 14:03:59.034314 (+ 153us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":83,"threads_started":1}
I20260429 14:03:59.046007 5874 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:03:59.046695 5874 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:03:59.055111 5874 catalog_manager.cc:6044] T 00000000000000000000000000000000 P df0fc0697e94470485d6496f41fa9021: Generated new TSK 0
I20260429 14:03:59.055302 5874 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:03:59.111385 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.193:0
--local_ip_for_outbound_sockets=127.2.197.193
--webserver_interface=127.2.197.193
--webserver_port=0
--tserver_master_addrs=127.2.197.254:34587
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:39855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:03:59.201370 5894 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:03:59.201573 5894 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:03:59.201617 5894 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:03:59.203269 5894 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:03:59.203325 5894 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:03:59.203356 5894 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.193
I20260429 14:03:59.205071 5894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:39855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.197.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:34587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.193
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:03:59.205324 5894 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:03:59.205530 5894 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:03:59.208460 5899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:03:59.208509 5902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:03:59.208457 5900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:03:59.208954 5894 server_base.cc:1061] running on GCE node
I20260429 14:03:59.209160 5894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:03:59.209338 5894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:03:59.210505 5894 hybrid_clock.cc:648] HybridClock initialized: now 1777471439210484 us; error 36 us; skew 500 ppm
Apr 29 14:03:59 dist-test-slave-8n77 krb5kdc[5782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471439, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:03:59.213001 5894 init.cc:377] Logged in from keytab as kudu/127.2.197.193@KRBTEST.COM (short username kudu)
I20260429 14:03:59.213574 5894 webserver.cc:492] Webserver started at http://127.2.197.193:35443/ using document root <none> and password file <none>
I20260429 14:03:59.213779 5894 fs_manager.cc:362] Metadata directory not provided
I20260429 14:03:59.213822 5894 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:03:59.213938 5894 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:03:59.214957 5894 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-0/data/instance:
uuid: "c9b9b4b5c7364de090b051ff0c33218e"
format_stamp: "Formatted at 2026-04-29 14:03:59 on dist-test-slave-8n77"
server_key: "080ea84b47a1d9252ee3844a2531f609"
server_key_iv: "8d7be33c2c1316ec81643d246ea631b8"
server_key_version: "encryptionkey@0"
I20260429 14:03:59.215286 5894 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance:
uuid: "c9b9b4b5c7364de090b051ff0c33218e"
format_stamp: "Formatted at 2026-04-29 14:03:59 on dist-test-slave-8n77"
server_key: "080ea84b47a1d9252ee3844a2531f609"
server_key_iv: "8d7be33c2c1316ec81643d246ea631b8"
server_key_version: "encryptionkey@0"
I20260429 14:03:59.216591 5894 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260429 14:03:59.217432 5909 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:59.217656 5894 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:03:59.217736 5894 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-0/wal
uuid: "c9b9b4b5c7364de090b051ff0c33218e"
format_stamp: "Formatted at 2026-04-29 14:03:59 on dist-test-slave-8n77"
server_key: "080ea84b47a1d9252ee3844a2531f609"
server_key_iv: "8d7be33c2c1316ec81643d246ea631b8"
server_key_version: "encryptionkey@0"
I20260429 14:03:59.217818 5894 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:03:59.241240 5894 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:03:59.243369 5894 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:03:59.243541 5894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:03:59.243794 5894 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:03:59.244119 5894 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:03:59.244174 5894 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:59.244217 5894 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:03:59.244241 5894 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:59.250438 5894 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.193:35547
I20260429 14:03:59.250476 6022 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.193:35547 every 8 connection(s)
I20260429 14:03:59.250803 5894 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
Apr 29 14:03:59 dist-test-slave-8n77 krb5kdc[5782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471439, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:03:59.256956 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 5894
I20260429 14:03:59.257040 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance
I20260429 14:03:59.257229 2839 external_mini_cluster.cc:1468] Setting key 222482616d8bf30f04c9ae600f1bdc23
I20260429 14:03:59.259750 5880 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:59.251560 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:54827 (local address 127.2.197.254:34587)
0429 14:03:59.251668 (+ 108us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:59.251671 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:59.251856 (+ 185us) server_negotiation.cc:408] Connection header received
0429 14:03:59.252297 (+ 441us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:59.252299 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:59.252312 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:59.252368 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:59.253103 (+ 735us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:59.253616 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:59.254202 (+ 586us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:59.254430 (+ 228us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:59.256661 (+ 2231us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:59.256665 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:59.256666 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:59.256673 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:59.258457 (+ 1784us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:59.258877 (+ 420us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:59.258878 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:59.258878 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:59.258904 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:59.259103 (+ 199us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:59.259104 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:59.259104 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:59.259210 (+ 106us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:59.259284 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:59.259549 (+ 265us) server_negotiation.cc:300] Negotiation successful
0429 14:03:59.259656 (+ 107us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260429 14:03:59.259891 6025 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:59.251532 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:34587 (local address 127.2.197.193:54827)
0429 14:03:59.251736 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:59.251764 (+ 28us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:59.252221 (+ 457us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:59.252429 (+ 208us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:59.252432 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:59.252729 (+ 297us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:03:59.253006 (+ 277us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:59.253010 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:59.253703 (+ 693us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:59.253704 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:59.254101 (+ 397us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:59.254103 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:59.254193 (+ 90us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:59.255206 (+ 1013us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:03:59.255215 (+ 9us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:03:59.256560 (+ 1345us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:03:59.258552 (+ 1992us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:59.258560 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:59.258562 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:59.258784 (+ 222us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:59.258972 (+ 188us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:59.258973 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:59.258973 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:59.259038 (+ 65us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:59.259274 (+ 236us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:03:59.259279 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:03:59.259403 (+ 124us) client_negotiation.cc:770] Sending connection context
0429 14:03:59.259512 (+ 109us) client_negotiation.cc:241] Negotiation successful
0429 14:03:59.259656 (+ 144us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":128,"threads_started":1}
I20260429 14:03:59.260468 6023 heartbeater.cc:344] Connected to a master server at 127.2.197.254:34587
I20260429 14:03:59.260574 6023 heartbeater.cc:461] Registering TS with master...
I20260429 14:03:59.260783 6023 heartbeater.cc:507] Master 127.2.197.254:34587 requested a full tablet report, sending...
I20260429 14:03:59.261379 5830 ts_manager.cc:194] Registered new tserver with Master: c9b9b4b5c7364de090b051ff0c33218e (127.2.197.193:35547)
I20260429 14:03:59.262437 5830 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.193@KRBTEST.COM'} at 127.2.197.193:54827
WARNING: no policy specified for kudu/127.2.197.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:03:59.315014 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.194:0
--local_ip_for_outbound_sockets=127.2.197.194
--webserver_interface=127.2.197.194
--webserver_port=0
--tserver_master_addrs=127.2.197.254:34587
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:39855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:03:59.395164 6030 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:03:59.395382 6030 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:03:59.395411 6030 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:03:59.397006 6030 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:03:59.397097 6030 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:03:59.397130 6030 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.194
I20260429 14:03:59.398942 6030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:39855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.197.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:34587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.194
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:03:59.399185 6030 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:03:59.399370 6030 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:03:59.402169 6038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:03:59.402138 6035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:03:59.402161 6036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:03:59.402463 6030 server_base.cc:1061] running on GCE node
I20260429 14:03:59.402817 6030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:03:59.403146 6030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:03:59.404369 6030 hybrid_clock.cc:648] HybridClock initialized: now 1777471439404349 us; error 27 us; skew 500 ppm
Apr 29 14:03:59 dist-test-slave-8n77 krb5kdc[5782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471439, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:03:59.407063 6030 init.cc:377] Logged in from keytab as kudu/127.2.197.194@KRBTEST.COM (short username kudu)
I20260429 14:03:59.407675 6030 webserver.cc:492] Webserver started at http://127.2.197.194:42647/ using document root <none> and password file <none>
I20260429 14:03:59.407884 6030 fs_manager.cc:362] Metadata directory not provided
I20260429 14:03:59.407944 6030 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:03:59.408049 6030 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:03:59.408998 6030 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-1/data/instance:
uuid: "6f686addbc394ad1ae43f97b034ba84a"
format_stamp: "Formatted at 2026-04-29 14:03:59 on dist-test-slave-8n77"
server_key: "89fc293946c34a4e6bd63e464e859f7f"
server_key_iv: "1286b15a6ccae91ed33c5a4561fa7ce2"
server_key_version: "encryptionkey@0"
I20260429 14:03:59.409332 6030 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance:
uuid: "6f686addbc394ad1ae43f97b034ba84a"
format_stamp: "Formatted at 2026-04-29 14:03:59 on dist-test-slave-8n77"
server_key: "89fc293946c34a4e6bd63e464e859f7f"
server_key_iv: "1286b15a6ccae91ed33c5a4561fa7ce2"
server_key_version: "encryptionkey@0"
I20260429 14:03:59.410660 6030 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260429 14:03:59.411405 6045 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:59.411646 6030 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:03:59.411748 6030 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-1/wal
uuid: "6f686addbc394ad1ae43f97b034ba84a"
format_stamp: "Formatted at 2026-04-29 14:03:59 on dist-test-slave-8n77"
server_key: "89fc293946c34a4e6bd63e464e859f7f"
server_key_iv: "1286b15a6ccae91ed33c5a4561fa7ce2"
server_key_version: "encryptionkey@0"
I20260429 14:03:59.411846 6030 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:03:59.419713 6030 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:03:59.421752 6030 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:03:59.421912 6030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:03:59.422135 6030 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:03:59.422523 6030 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:03:59.422585 6030 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:59.422621 6030 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:03:59.422670 6030 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:59.428917 6030 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.194:39753
I20260429 14:03:59.428980 6158 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.194:39753 every 8 connection(s)
I20260429 14:03:59.429315 6030 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
I20260429 14:03:59.429853 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 6030
I20260429 14:03:59.429960 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance
I20260429 14:03:59.430141 2839 external_mini_cluster.cc:1468] Setting key a3d603136ce9606441fc146c64afb555
Apr 29 14:03:59 dist-test-slave-8n77 krb5kdc[5782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471439, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.197.195@KRBTEST.COM; defaulting to no policy
I20260429 14:03:59.438699 5880 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:59.430180 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:58997 (local address 127.2.197.254:34587)
0429 14:03:59.430229 (+ 49us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:59.430231 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:59.430549 (+ 318us) server_negotiation.cc:408] Connection header received
0429 14:03:59.431145 (+ 596us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:59.431146 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:59.431158 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:59.431215 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:59.431989 (+ 774us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:59.432609 (+ 620us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:59.433187 (+ 578us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:59.433391 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:59.435678 (+ 2287us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:59.435683 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:59.435684 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:59.435692 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:59.437312 (+ 1620us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:59.437737 (+ 425us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:59.437738 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:59.437739 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:59.437765 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:59.437964 (+ 199us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:59.437965 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:59.437966 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:59.438092 (+ 126us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:59.438169 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:59.438552 (+ 383us) server_negotiation.cc:300] Negotiation successful
0429 14:03:59.438633 (+ 81us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260429 14:03:59.438813 6161 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:59.430169 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:34587 (local address 127.2.197.194:58997)
0429 14:03:59.430432 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:59.430468 (+ 36us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:59.431038 (+ 570us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:59.431270 (+ 232us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:59.431279 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:59.431571 (+ 292us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:03:59.431838 (+ 267us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:59.431841 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:59.432716 (+ 875us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:59.432717 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:59.433099 (+ 382us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:59.433100 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:59.433186 (+ 86us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:59.434100 (+ 914us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:03:59.434109 (+ 9us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:03:59.435574 (+ 1465us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:03:59.437418 (+ 1844us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:59.437423 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:59.437425 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:59.437644 (+ 219us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:59.437844 (+ 200us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:59.437845 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:59.437845 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:59.437902 (+ 57us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:59.438203 (+ 301us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:03:59.438210 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:03:59.438396 (+ 186us) client_negotiation.cc:770] Sending connection context
0429 14:03:59.438498 (+ 102us) client_negotiation.cc:241] Negotiation successful
0429 14:03:59.438621 (+ 123us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":72,"threads_started":1}
I20260429 14:03:59.439332 6159 heartbeater.cc:344] Connected to a master server at 127.2.197.254:34587
I20260429 14:03:59.439446 6159 heartbeater.cc:461] Registering TS with master...
I20260429 14:03:59.439682 6159 heartbeater.cc:507] Master 127.2.197.254:34587 requested a full tablet report, sending...
I20260429 14:03:59.440128 5830 ts_manager.cc:194] Registered new tserver with Master: 6f686addbc394ad1ae43f97b034ba84a (127.2.197.194:39753)
I20260429 14:03:59.440559 5830 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.194@KRBTEST.COM'} at 127.2.197.194:58997
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:03:59.483574 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.195:0
--local_ip_for_outbound_sockets=127.2.197.195
--webserver_interface=127.2.197.195
--webserver_port=0
--tserver_master_addrs=127.2.197.254:34587
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:39855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:03:59.562577 6166 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:03:59.562767 6166 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:03:59.562789 6166 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:03:59.564307 6166 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:03:59.564363 6166 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:03:59.564391 6166 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.195
I20260429 14:03:59.566072 6166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:39855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.197.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:34587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.195
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:03:59.566342 6166 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:03:59.566550 6166 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:03:59.569605 6172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:03:59.569362 6174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:03:59.569375 6171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:03:59.570251 6166 server_base.cc:1061] running on GCE node
I20260429 14:03:59.570513 6166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:03:59.570782 6166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:03:59.571939 6166 hybrid_clock.cc:648] HybridClock initialized: now 1777471439571923 us; error 43 us; skew 500 ppm
Apr 29 14:03:59 dist-test-slave-8n77 krb5kdc[5782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471439, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:03:59.574626 6166 init.cc:377] Logged in from keytab as kudu/127.2.197.195@KRBTEST.COM (short username kudu)
I20260429 14:03:59.575239 6166 webserver.cc:492] Webserver started at http://127.2.197.195:33133/ using document root <none> and password file <none>
I20260429 14:03:59.575440 6166 fs_manager.cc:362] Metadata directory not provided
I20260429 14:03:59.575506 6166 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:03:59.575615 6166 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:03:59.576560 6166 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-2/data/instance:
uuid: "a92319ff791c4d71a057bdc0b0d00cd4"
format_stamp: "Formatted at 2026-04-29 14:03:59 on dist-test-slave-8n77"
server_key: "1ac2b121a7e7f6e39853212479a42b69"
server_key_iv: "15dba48e034774d4aab08c17750a87b3"
server_key_version: "encryptionkey@0"
I20260429 14:03:59.576951 6166 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance:
uuid: "a92319ff791c4d71a057bdc0b0d00cd4"
format_stamp: "Formatted at 2026-04-29 14:03:59 on dist-test-slave-8n77"
server_key: "1ac2b121a7e7f6e39853212479a42b69"
server_key_iv: "15dba48e034774d4aab08c17750a87b3"
server_key_version: "encryptionkey@0"
I20260429 14:03:59.578361 6166 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260429 14:03:59.579213 6181 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:59.579432 6166 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:03:59.579531 6166 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-2/wal
uuid: "a92319ff791c4d71a057bdc0b0d00cd4"
format_stamp: "Formatted at 2026-04-29 14:03:59 on dist-test-slave-8n77"
server_key: "1ac2b121a7e7f6e39853212479a42b69"
server_key_iv: "15dba48e034774d4aab08c17750a87b3"
server_key_version: "encryptionkey@0"
I20260429 14:03:59.579605 6166 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:03:59.592260 6166 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:03:59.594219 6166 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:03:59.594437 6166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:03:59.594727 6166 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:03:59.595041 6166 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:03:59.595095 6166 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:59.595139 6166 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:03:59.595167 6166 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:03:59.601765 6166 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.195:39767
I20260429 14:03:59.601876 6294 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.195:39767 every 8 connection(s)
I20260429 14:03:59.602351 6166 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
I20260429 14:03:59.608484 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 6166
I20260429 14:03:59.608688 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance
I20260429 14:03:59.608908 2839 external_mini_cluster.cc:1468] Setting key 30e89b0b8dcddcc9b2790b0e538e0143
Apr 29 14:03:59 dist-test-slave-8n77 krb5kdc[5782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471439, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:03:59.613665 5880 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:59.603562 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:34377 (local address 127.2.197.254:34587)
0429 14:03:59.603644 (+ 82us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:59.603646 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:59.603750 (+ 104us) server_negotiation.cc:408] Connection header received
0429 14:03:59.604319 (+ 569us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:59.604321 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:59.604337 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:59.604443 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:59.605396 (+ 953us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:59.606135 (+ 739us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:03:59.606890 (+ 755us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:03:59.607030 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:59.610351 (+ 3321us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:03:59.610358 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:03:59.610359 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:03:59.610371 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:03:59.612273 (+ 1902us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:59.612783 (+ 510us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:59.612791 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:59.612792 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:59.612825 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:03:59.613027 (+ 202us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:03:59.613028 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:03:59.613028 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:03:59.613137 (+ 109us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:03:59.613216 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0429 14:03:59.613471 (+ 255us) server_negotiation.cc:300] Negotiation successful
0429 14:03:59.613572 (+ 101us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260429 14:03:59.613842 6297 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:03:59.603435 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:34587 (local address 127.2.197.195:34377)
0429 14:03:59.603601 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0429 14:03:59.603648 (+ 47us) client_negotiation.cc:175] Beginning negotiation
0429 14:03:59.604141 (+ 493us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:03:59.604529 (+ 388us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:03:59.604533 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:03:59.604860 (+ 327us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:03:59.605125 (+ 265us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:59.605128 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:59.606272 (+ 1144us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:03:59.606335 (+ 63us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:03:59.606780 (+ 445us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:03:59.606782 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:03:59.606910 (+ 128us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:03:59.607914 (+ 1004us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:03:59.607923 (+ 9us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:03:59.610206 (+ 2283us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:03:59.612402 (+ 2196us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:59.612411 (+ 9us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:59.612414 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:59.612694 (+ 280us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:59.612892 (+ 198us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:03:59.612893 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:03:59.612893 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:03:59.612964 (+ 71us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:03:59.613203 (+ 239us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:03:59.613208 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:03:59.613338 (+ 130us) client_negotiation.cc:770] Sending connection context
0429 14:03:59.613453 (+ 115us) client_negotiation.cc:241] Negotiation successful
0429 14:03:59.613614 (+ 161us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":98,"threads_started":1}
I20260429 14:03:59.614530 6295 heartbeater.cc:344] Connected to a master server at 127.2.197.254:34587
I20260429 14:03:59.614634 6295 heartbeater.cc:461] Registering TS with master...
I20260429 14:03:59.614845 6295 heartbeater.cc:507] Master 127.2.197.254:34587 requested a full tablet report, sending...
I20260429 14:03:59.615370 5830 ts_manager.cc:194] Registered new tserver with Master: a92319ff791c4d71a057bdc0b0d00cd4 (127.2.197.195:39767)
I20260429 14:03:59.615816 5830 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.195@KRBTEST.COM'} at 127.2.197.195:34377
I20260429 14:03:59.622045 2839 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260429 14:03:59.723153 5880 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:03:59.720786 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44350 (local address 127.2.197.254:34587)
0429 14:03:59.720854 (+ 68us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:59.720857 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:59.721077 (+ 220us) server_negotiation.cc:408] Connection header received
0429 14:03:59.721491 (+ 414us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:59.721493 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:59.721503 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:59.721563 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:59.723102 (+ 1539us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44350: BlockingRecv error: recv got EOF from 127.0.0.1:44350 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260429 14:03:59.723997 5880 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:03:59.723407 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44352 (local address 127.2.197.254:34587)
0429 14:03:59.723444 (+ 37us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:59.723446 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:59.723508 (+ 62us) server_negotiation.cc:408] Connection header received
0429 14:03:59.723574 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:59.723575 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:59.723586 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:59.723631 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:59.723963 (+ 332us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44352: BlockingRecv error: recv got EOF from 127.0.0.1:44352 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260429 14:03:59.724581 5880 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:03:59.724142 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44354 (local address 127.2.197.254:34587)
0429 14:03:59.724194 (+ 52us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:59.724195 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:59.724204 (+ 9us) server_negotiation.cc:408] Connection header received
0429 14:03:59.724229 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:59.724229 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:59.724236 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:59.724285 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:59.724551 (+ 266us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44354: BlockingRecv error: recv got EOF from 127.0.0.1:44354 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260429 14:03:59.726549 5880 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:03:59.725837 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44368 (local address 127.2.197.254:34587)
0429 14:03:59.725903 (+ 66us) server_negotiation.cc:207] Beginning negotiation
0429 14:03:59.725906 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:03:59.725919 (+ 13us) server_negotiation.cc:408] Connection header received
0429 14:03:59.725936 (+ 17us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:03:59.725937 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:03:59.725944 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:03:59.725998 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:03:59.726427 (+ 429us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44368: BlockingRecv error: recv got EOF from 127.0.0.1:44368 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260429 14:03:59.782682 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 5894
I20260429 14:03:59.788229 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 6030
I20260429 14:03:59.793090 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 6166
I20260429 14:03:59.797822 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 5798
2026-04-29T14:03:59Z chronyd exiting
[ OK ] SecurityITest.TestRebalanceReportUnauthorized (3102 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:03:59 dist-test-slave-8n77 krb5kdc[6318](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:03:59 dist-test-slave-8n77 krb5kdc[6318](info): set up 2 sockets
Apr 29 14:03:59 dist-test-slave-8n77 krb5kdc[6318](info): commencing operation
krb5kdc: starting...
W20260429 14:04:01.877869 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.058s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:04:01 dist-test-slave-8n77 krb5kdc[6318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:04:01Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:04:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:04:02.038102 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:41523
--webserver_interface=127.2.197.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:43101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:41523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:04:02.118010 6334 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:02.118227 6334 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:02.118250 6334 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:02.119875 6334 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:04:02.119935 6334 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:02.119951 6334 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:02.119963 6334 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:04:02.119976 6334 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:04:02.121676 6334 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:43101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:41523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:41523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:02.121956 6334 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:02.122177 6334 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:02.125324 6339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:02.125388 6342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:02.125540 6334 server_base.cc:1061] running on GCE node
W20260429 14:04:02.125622 6340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:02.125841 6334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:02.126102 6334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:02.127283 6334 hybrid_clock.cc:648] HybridClock initialized: now 1777471442127273 us; error 66 us; skew 500 ppm
Apr 29 14:04:02 dist-test-slave-8n77 krb5kdc[6318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471442, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:02.129846 6334 init.cc:377] Logged in from keytab as kudu/127.2.197.254@KRBTEST.COM (short username kudu)
I20260429 14:04:02.130522 6334 webserver.cc:492] Webserver started at http://127.2.197.254:46837/ using document root <none> and password file <none>
I20260429 14:04:02.130740 6334 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:02.130805 6334 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:02.130930 6334 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:04:02.131909 6334 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "ed8b34da86144c39a049612be1770809"
format_stamp: "Formatted at 2026-04-29 14:04:02 on dist-test-slave-8n77"
server_key: "880fba7e2db5c20c9643cb231dd5fd3b"
server_key_iv: "c71847bce58a68035bc4eab0caccff9c"
server_key_version: "encryptionkey@0"
I20260429 14:04:02.132232 6334 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "ed8b34da86144c39a049612be1770809"
format_stamp: "Formatted at 2026-04-29 14:04:02 on dist-test-slave-8n77"
server_key: "880fba7e2db5c20c9643cb231dd5fd3b"
server_key_iv: "c71847bce58a68035bc4eab0caccff9c"
server_key_version: "encryptionkey@0"
I20260429 14:04:02.133512 6334 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260429 14:04:02.134380 6349 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:02.134631 6334 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:02.134717 6334 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "ed8b34da86144c39a049612be1770809"
format_stamp: "Formatted at 2026-04-29 14:04:02 on dist-test-slave-8n77"
server_key: "880fba7e2db5c20c9643cb231dd5fd3b"
server_key_iv: "c71847bce58a68035bc4eab0caccff9c"
server_key_version: "encryptionkey@0"
I20260429 14:04:02.134805 6334 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:02.162864 6334 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:02.172257 6334 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:02.172416 6334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:02.179303 6334 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:41523
I20260429 14:04:02.179332 6401 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:41523 every 8 connection(s)
I20260429 14:04:02.179761 6334 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:04:02.180491 6402 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:02.183517 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 6334
I20260429 14:04:02.183643 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:04:02.183873 2839 external_mini_cluster.cc:1468] Setting key a2259054079fe826bc69e10937ffd711
I20260429 14:04:02.185112 6402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed8b34da86144c39a049612be1770809: Bootstrap starting.
I20260429 14:04:02.186133 6402 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed8b34da86144c39a049612be1770809: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:02.186636 6402 log.cc:826] T 00000000000000000000000000000000 P ed8b34da86144c39a049612be1770809: Log is configured to *not* fsync() on all Append() calls
I20260429 14:04:02.188072 6402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed8b34da86144c39a049612be1770809: No bootstrap required, opened a new log
I20260429 14:04:02.190548 6402 raft_consensus.cc:359] T 00000000000000000000000000000000 P ed8b34da86144c39a049612be1770809 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8b34da86144c39a049612be1770809" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 41523 } }
I20260429 14:04:02.190758 6402 raft_consensus.cc:385] T 00000000000000000000000000000000 P ed8b34da86144c39a049612be1770809 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:02.190796 6402 raft_consensus.cc:740] T 00000000000000000000000000000000 P ed8b34da86144c39a049612be1770809 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed8b34da86144c39a049612be1770809, State: Initialized, Role: FOLLOWER
I20260429 14:04:02.190974 6402 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed8b34da86144c39a049612be1770809 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8b34da86144c39a049612be1770809" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 41523 } }
I20260429 14:04:02.191263 6402 raft_consensus.cc:399] T 00000000000000000000000000000000 P ed8b34da86144c39a049612be1770809 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:04:02.191365 6402 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed8b34da86144c39a049612be1770809 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:04:02.191460 6402 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ed8b34da86144c39a049612be1770809 [term 0 FOLLOWER]: Advancing to term 1
Apr 29 14:04:02 dist-test-slave-8n77 krb5kdc[6318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:04:02.192322 6402 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed8b34da86144c39a049612be1770809 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8b34da86144c39a049612be1770809" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 41523 } }
I20260429 14:04:02.192476 6402 leader_election.cc:304] T 00000000000000000000000000000000 P ed8b34da86144c39a049612be1770809 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed8b34da86144c39a049612be1770809; no voters:
I20260429 14:04:02.192699 6402 leader_election.cc:290] T 00000000000000000000000000000000 P ed8b34da86144c39a049612be1770809 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:04:02.192940 6407 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ed8b34da86144c39a049612be1770809 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:04:02.193009 6402 sys_catalog.cc:565] T 00000000000000000000000000000000 P ed8b34da86144c39a049612be1770809 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:04:02.193148 6407 raft_consensus.cc:697] T 00000000000000000000000000000000 P ed8b34da86144c39a049612be1770809 [term 1 LEADER]: Becoming Leader. State: Replica: ed8b34da86144c39a049612be1770809, State: Running, Role: LEADER
I20260429 14:04:02.193265 6407 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed8b34da86144c39a049612be1770809 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8b34da86144c39a049612be1770809" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 41523 } }
I20260429 14:04:02.194360 6409 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed8b34da86144c39a049612be1770809 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed8b34da86144c39a049612be1770809. Latest consensus state: current_term: 1 leader_uuid: "ed8b34da86144c39a049612be1770809" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8b34da86144c39a049612be1770809" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 41523 } } }
I20260429 14:04:02.194614 6409 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed8b34da86144c39a049612be1770809 [sys.catalog]: This master's current role is: LEADER
I20260429 14:04:02.194887 6416 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:04:02.195418 6416 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:04:02.195683 6408 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed8b34da86144c39a049612be1770809 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed8b34da86144c39a049612be1770809" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8b34da86144c39a049612be1770809" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 41523 } } }
I20260429 14:04:02.196071 6408 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed8b34da86144c39a049612be1770809 [sys.catalog]: This master's current role is: LEADER
I20260429 14:04:02.197067 6416 catalog_manager.cc:1357] Generated new cluster ID: 335125926aa643e298470a7f0216f6b8
I20260429 14:04:02.197117 6416 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:04:02.197450 6405 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:02.184710 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:39242 (local address 127.2.197.254:41523)
0429 14:04:02.185015 (+ 305us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:02.185028 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:02.185054 (+ 26us) server_negotiation.cc:408] Connection header received
0429 14:04:02.185554 (+ 500us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:02.185566 (+ 12us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:02.185629 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:02.185977 (+ 348us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:02.187142 (+ 1165us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:02.188835 (+ 1693us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:02.189760 (+ 925us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:02.190088 (+ 328us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:02.192462 (+ 2374us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:02.192475 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:02.192475 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:02.192493 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:02.195355 (+ 2862us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:02.195857 (+ 502us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:02.195860 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:02.195861 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:02.195930 (+ 69us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:02.196156 (+ 226us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:02.196157 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:02.196157 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:02.196410 (+ 253us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:02.196570 (+ 160us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:02.197145 (+ 575us) server_negotiation.cc:300] Negotiation successful
0429 14:04:02.197281 (+ 136us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":131,"threads_started":1}
I20260429 14:04:02.198382 2839 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260429 14:04:02.217355 6416 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:04:02.217512 6405 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:02.213002 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:39254 (local address 127.2.197.254:41523)
0429 14:04:02.213052 (+ 50us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:02.213055 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:02.213190 (+ 135us) server_negotiation.cc:408] Connection header received
0429 14:04:02.213248 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:02.213249 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:02.213263 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:02.213348 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:02.214368 (+ 1020us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:02.215159 (+ 791us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:02.216047 (+ 888us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:02.216243 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:02.216910 (+ 667us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:02.216915 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:02.216920 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:04:02.217198 (+ 278us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:02.217224 (+ 26us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:04:02.217230 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:02.217310 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:02.217396 (+ 86us) server_negotiation.cc:300] Negotiation successful
0429 14:04:02.217426 (+ 30us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260429 14:04:02.218169 6416 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:04:02.225692 6416 catalog_manager.cc:6044] T 00000000000000000000000000000000 P ed8b34da86144c39a049612be1770809: Generated new TSK 0
I20260429 14:04:02.225966 6416 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 14:04:02.231335 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 6334
2026-04-29T14:04:02Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2432 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:04:02 dist-test-slave-8n77 krb5kdc[6435](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:04:02 dist-test-slave-8n77 krb5kdc[6435](info): set up 2 sockets
Apr 29 14:04:02 dist-test-slave-8n77 krb5kdc[6435](info): commencing operation
krb5kdc: starting...
W20260429 14:04:04.284830 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.033s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:04:04 dist-test-slave-8n77 krb5kdc[6435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:04:04Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:04:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:04:04.441998 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:43919
--webserver_interface=127.2.197.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:39937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:43919
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:04:04.526610 6451 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:04.526845 6451 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:04.526872 6451 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:04.529143 6451 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:04:04.529253 6451 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:04.529269 6451 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:04.529284 6451 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:04:04.529296 6451 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:04:04.532204 6451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:39937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:43919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:43919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:04.532563 6451 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:04.532790 6451 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260429 14:04:04.535673 6451 server_base.cc:1061] running on GCE node
W20260429 14:04:04.535679 6456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:04.535692 6457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:04.535692 6459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:04.536126 6451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:04.536393 6451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:04.537587 6451 hybrid_clock.cc:648] HybridClock initialized: now 1777471444537569 us; error 56 us; skew 500 ppm
Apr 29 14:04:04 dist-test-slave-8n77 krb5kdc[6435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471444, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:04.540578 6451 init.cc:377] Logged in from keytab as kudu/127.2.197.254@KRBTEST.COM (short username kudu)
I20260429 14:04:04.541393 6451 webserver.cc:492] Webserver started at http://127.2.197.254:41969/ using document root <none> and password file <none>
I20260429 14:04:04.541741 6451 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:04.541851 6451 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:04.541972 6451 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:04:04.543339 6451 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "07757715694b4e1fa38570c723ef5959"
format_stamp: "Formatted at 2026-04-29 14:04:04 on dist-test-slave-8n77"
server_key: "1f6bfbd9468fa02fa1c94336b344a66c"
server_key_iv: "ea40c1ef15b43e6e228346528781b909"
server_key_version: "encryptionkey@0"
I20260429 14:04:04.543957 6451 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "07757715694b4e1fa38570c723ef5959"
format_stamp: "Formatted at 2026-04-29 14:04:04 on dist-test-slave-8n77"
server_key: "1f6bfbd9468fa02fa1c94336b344a66c"
server_key_iv: "ea40c1ef15b43e6e228346528781b909"
server_key_version: "encryptionkey@0"
I20260429 14:04:04.545547 6451 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260429 14:04:04.546439 6466 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:04.546855 6451 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260429 14:04:04.546947 6451 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "07757715694b4e1fa38570c723ef5959"
format_stamp: "Formatted at 2026-04-29 14:04:04 on dist-test-slave-8n77"
server_key: "1f6bfbd9468fa02fa1c94336b344a66c"
server_key_iv: "ea40c1ef15b43e6e228346528781b909"
server_key_version: "encryptionkey@0"
I20260429 14:04:04.547032 6451 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:04.574854 6451 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:04.576824 6451 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:04.576998 6451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:04.581571 6451 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:43919
I20260429 14:04:04.581729 6518 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:43919 every 8 connection(s)
I20260429 14:04:04.581982 6451 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:04:04.582924 6519 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:04.585361 6519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07757715694b4e1fa38570c723ef5959: Bootstrap starting.
I20260429 14:04:04.585979 6519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07757715694b4e1fa38570c723ef5959: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:04.586246 6519 log.cc:826] T 00000000000000000000000000000000 P 07757715694b4e1fa38570c723ef5959: Log is configured to *not* fsync() on all Append() calls
I20260429 14:04:04.587134 6519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07757715694b4e1fa38570c723ef5959: No bootstrap required, opened a new log
I20260429 14:04:04.588421 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 6451
I20260429 14:04:04.588523 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:04:04.588513 6519 raft_consensus.cc:359] T 00000000000000000000000000000000 P 07757715694b4e1fa38570c723ef5959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07757715694b4e1fa38570c723ef5959" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43919 } }
I20260429 14:04:04.588693 6519 raft_consensus.cc:385] T 00000000000000000000000000000000 P 07757715694b4e1fa38570c723ef5959 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:04.588755 6519 raft_consensus.cc:740] T 00000000000000000000000000000000 P 07757715694b4e1fa38570c723ef5959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07757715694b4e1fa38570c723ef5959, State: Initialized, Role: FOLLOWER
I20260429 14:04:04.588819 2839 external_mini_cluster.cc:1468] Setting key 3541d1f36ca58a058be3691c996e8c46
I20260429 14:04:04.588918 6519 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07757715694b4e1fa38570c723ef5959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07757715694b4e1fa38570c723ef5959" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43919 } }
I20260429 14:04:04.589011 6519 raft_consensus.cc:399] T 00000000000000000000000000000000 P 07757715694b4e1fa38570c723ef5959 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:04:04.589056 6519 raft_consensus.cc:493] T 00000000000000000000000000000000 P 07757715694b4e1fa38570c723ef5959 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:04:04.589097 6519 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 07757715694b4e1fa38570c723ef5959 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:04.589860 6519 raft_consensus.cc:515] T 00000000000000000000000000000000 P 07757715694b4e1fa38570c723ef5959 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07757715694b4e1fa38570c723ef5959" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43919 } }
I20260429 14:04:04.589985 6519 leader_election.cc:304] T 00000000000000000000000000000000 P 07757715694b4e1fa38570c723ef5959 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07757715694b4e1fa38570c723ef5959; no voters:
I20260429 14:04:04.590193 6519 leader_election.cc:290] T 00000000000000000000000000000000 P 07757715694b4e1fa38570c723ef5959 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:04:04.590251 6524 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 07757715694b4e1fa38570c723ef5959 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:04:04.590487 6524 raft_consensus.cc:697] T 00000000000000000000000000000000 P 07757715694b4e1fa38570c723ef5959 [term 1 LEADER]: Becoming Leader. State: Replica: 07757715694b4e1fa38570c723ef5959, State: Running, Role: LEADER
I20260429 14:04:04.590536 6519 sys_catalog.cc:565] T 00000000000000000000000000000000 P 07757715694b4e1fa38570c723ef5959 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:04:04.590618 6524 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07757715694b4e1fa38570c723ef5959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07757715694b4e1fa38570c723ef5959" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43919 } }
I20260429 14:04:04.590922 6525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07757715694b4e1fa38570c723ef5959 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07757715694b4e1fa38570c723ef5959" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07757715694b4e1fa38570c723ef5959" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43919 } } }
I20260429 14:04:04.590914 6524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07757715694b4e1fa38570c723ef5959 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07757715694b4e1fa38570c723ef5959. Latest consensus state: current_term: 1 leader_uuid: "07757715694b4e1fa38570c723ef5959" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07757715694b4e1fa38570c723ef5959" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43919 } } }
I20260429 14:04:04.591051 6525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07757715694b4e1fa38570c723ef5959 [sys.catalog]: This master's current role is: LEADER
I20260429 14:04:04.591060 6524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07757715694b4e1fa38570c723ef5959 [sys.catalog]: This master's current role is: LEADER
W20260429 14:04:04.592631 6539 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 07757715694b4e1fa38570c723ef5959: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260429 14:04:04.592748 6539 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260429 14:04:04.592957 6530 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:04:04.593505 6530 catalog_manager.cc:1494] Initializing Kudu cluster ID...
Apr 29 14:04:04 dist-test-slave-8n77 krb5kdc[6435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:04:04.595808 6530 catalog_manager.cc:1357] Generated new cluster ID: 7b027451e5cf4d22816c1ba82d6fce9c
I20260429 14:04:04.595877 6530 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:04:04.600314 6523 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:04.589588 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:43202 (local address 127.2.197.254:43919)
0429 14:04:04.589869 (+ 281us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:04.589879 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:04.589902 (+ 23us) server_negotiation.cc:408] Connection header received
0429 14:04:04.590026 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:04.590033 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:04.590106 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:04.590266 (+ 160us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:04.591264 (+ 998us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:04.592645 (+ 1381us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:04.593663 (+ 1018us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:04.593904 (+ 241us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:04.596310 (+ 2406us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:04.596328 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:04.596328 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:04.596341 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:04.598455 (+ 2114us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:04.598854 (+ 399us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:04.598857 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:04.598858 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:04.598915 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:04.599079 (+ 164us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:04.599080 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:04.599080 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:04.599318 (+ 238us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:04.599443 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:04.599996 (+ 553us) server_negotiation.cc:300] Negotiation successful
0429 14:04:04.600130 (+ 134us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":183,"threads_started":1}
I20260429 14:04:04.623085 6530 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:04:04.623912 6530 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:04:04.639894 6530 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 07757715694b4e1fa38570c723ef5959: Generated new TSK 0
I20260429 14:04:04.640105 6530 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:04:04.699862 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.193:0
--local_ip_for_outbound_sockets=127.2.197.193
--webserver_interface=127.2.197.193
--webserver_port=0
--tserver_master_addrs=127.2.197.254:43919
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:39937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:04:04.783939 6547 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:04.784142 6547 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:04.784166 6547 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:04.785766 6547 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:04.785820 6547 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:04.785847 6547 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.193
I20260429 14:04:04.787496 6547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:39937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.197.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:43919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.193
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:04.787695 6547 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:04.787837 6547 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:04.790870 6555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:04.790870 6552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:04.791091 6547 server_base.cc:1061] running on GCE node
W20260429 14:04:04.791198 6553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:04.791393 6547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:04.791662 6547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:04.792879 6547 hybrid_clock.cc:648] HybridClock initialized: now 1777471444792858 us; error 40 us; skew 500 ppm
Apr 29 14:04:04 dist-test-slave-8n77 krb5kdc[6435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471444, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:04.795532 6547 init.cc:377] Logged in from keytab as kudu/127.2.197.193@KRBTEST.COM (short username kudu)
I20260429 14:04:04.796288 6547 webserver.cc:492] Webserver started at http://127.2.197.193:45643/ using document root <none> and password file <none>
I20260429 14:04:04.796577 6547 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:04.796687 6547 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:04.796828 6547 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:04:04.797827 6547 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-0/data/instance:
uuid: "c811f05ba0a1463296782d1ba07024e4"
format_stamp: "Formatted at 2026-04-29 14:04:04 on dist-test-slave-8n77"
server_key: "b8780b3e3339f1c10bd2ae1ac5ae4642"
server_key_iv: "dce26a0a8500d87e1b96e9b4e8c2e9e4"
server_key_version: "encryptionkey@0"
I20260429 14:04:04.798192 6547 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance:
uuid: "c811f05ba0a1463296782d1ba07024e4"
format_stamp: "Formatted at 2026-04-29 14:04:04 on dist-test-slave-8n77"
server_key: "b8780b3e3339f1c10bd2ae1ac5ae4642"
server_key_iv: "dce26a0a8500d87e1b96e9b4e8c2e9e4"
server_key_version: "encryptionkey@0"
I20260429 14:04:04.799572 6547 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:04:04.800302 6562 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:04.800585 6547 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260429 14:04:04.800678 6547 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-0/wal
uuid: "c811f05ba0a1463296782d1ba07024e4"
format_stamp: "Formatted at 2026-04-29 14:04:04 on dist-test-slave-8n77"
server_key: "b8780b3e3339f1c10bd2ae1ac5ae4642"
server_key_iv: "dce26a0a8500d87e1b96e9b4e8c2e9e4"
server_key_version: "encryptionkey@0"
I20260429 14:04:04.800762 6547 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:04.817611 6547 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:04.819660 6547 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:04.819828 6547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:04.820079 6547 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:04:04.820397 6547 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:04:04.820451 6547 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:04.820492 6547 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:04:04.820521 6547 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:04.826673 6547 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.193:34657
I20260429 14:04:04.826738 6675 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.193:34657 every 8 connection(s)
I20260429 14:04:04.827021 6547 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
Apr 29 14:04:04 dist-test-slave-8n77 krb5kdc[6435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471444, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:04:04.835506 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 6547
I20260429 14:04:04.835615 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance
I20260429 14:04:04.835656 6523 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:04.827759 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:38671 (local address 127.2.197.254:43919)
0429 14:04:04.827832 (+ 73us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:04.827834 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:04.828015 (+ 181us) server_negotiation.cc:408] Connection header received
0429 14:04:04.828454 (+ 439us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:04.828455 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:04.828469 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:04.828534 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:04.829298 (+ 764us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:04.829846 (+ 548us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:04.830530 (+ 684us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:04.830664 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:04.833054 (+ 2390us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:04.833058 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:04.833058 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:04.833065 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:04.834412 (+ 1347us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:04.834787 (+ 375us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:04.834788 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:04.834788 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:04.834808 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:04.834975 (+ 167us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:04.834975 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:04.834976 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:04.835064 (+ 88us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:04.835115 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:04.835453 (+ 338us) server_negotiation.cc:300] Negotiation successful
0429 14:04:04.835530 (+ 77us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260429 14:04:04.835814 2839 external_mini_cluster.cc:1468] Setting key 925221141913dbeb21f88430ef846c68
I20260429 14:04:04.835863 6678 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:04.827727 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:43919 (local address 127.2.197.193:38671)
0429 14:04:04.827892 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:04.827925 (+ 33us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:04.828377 (+ 452us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:04.828632 (+ 255us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:04.828636 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:04.828928 (+ 292us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:04.829178 (+ 250us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:04.829181 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:04.829954 (+ 773us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:04.829955 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:04.830411 (+ 456us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:04.830413 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:04.830506 (+ 93us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:04.831463 (+ 957us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:04.831472 (+ 9us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:04.832966 (+ 1494us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:04.834508 (+ 1542us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:04.834513 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:04.834515 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:04.834717 (+ 202us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:04.834866 (+ 149us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:04.834867 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:04.834867 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:04.834922 (+ 55us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:04.835145 (+ 223us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:04.835151 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:04.835275 (+ 124us) client_negotiation.cc:770] Sending connection context
0429 14:04:04.835446 (+ 171us) client_negotiation.cc:241] Negotiation successful
0429 14:04:04.835652 (+ 206us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":104,"threads_started":1}
I20260429 14:04:04.836386 6676 heartbeater.cc:344] Connected to a master server at 127.2.197.254:43919
I20260429 14:04:04.836563 6676 heartbeater.cc:461] Registering TS with master...
I20260429 14:04:04.836820 6676 heartbeater.cc:507] Master 127.2.197.254:43919 requested a full tablet report, sending...
I20260429 14:04:04.837324 6483 ts_manager.cc:194] Registered new tserver with Master: c811f05ba0a1463296782d1ba07024e4 (127.2.197.193:34657)
I20260429 14:04:04.838277 6483 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.193@KRBTEST.COM'} at 127.2.197.193:38671
WARNING: no policy specified for kudu/127.2.197.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:04:04.891009 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.194:0
--local_ip_for_outbound_sockets=127.2.197.194
--webserver_interface=127.2.197.194
--webserver_port=0
--tserver_master_addrs=127.2.197.254:43919
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:39937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:04:04.968669 6683 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:04.968861 6683 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:04.968886 6683 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:04.970484 6683 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:04.970538 6683 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:04.970566 6683 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.194
I20260429 14:04:04.972298 6683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:39937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.197.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:43919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.194
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:04.972501 6683 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:04.972719 6683 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260429 14:04:04.976080 6683 server_base.cc:1061] running on GCE node
W20260429 14:04:04.976109 6689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:04.976135 6691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:04.976260 6688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:04.976497 6683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:04.976764 6683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:04.977998 6683 hybrid_clock.cc:648] HybridClock initialized: now 1777471444977984 us; error 38 us; skew 500 ppm
Apr 29 14:04:04 dist-test-slave-8n77 krb5kdc[6435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471444, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:04.980762 6683 init.cc:377] Logged in from keytab as kudu/127.2.197.194@KRBTEST.COM (short username kudu)
I20260429 14:04:04.981765 6683 webserver.cc:492] Webserver started at http://127.2.197.194:40625/ using document root <none> and password file <none>
I20260429 14:04:04.982085 6683 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:04.982163 6683 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:04.982362 6683 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:04:04.983767 6683 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-1/data/instance:
uuid: "b308843a4f554d03a52e4f508c24679a"
format_stamp: "Formatted at 2026-04-29 14:04:04 on dist-test-slave-8n77"
server_key: "1a66aab9489ea60511696a93cf6e73e5"
server_key_iv: "7d104283976a483177aeb3b35660145d"
server_key_version: "encryptionkey@0"
I20260429 14:04:04.984198 6683 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance:
uuid: "b308843a4f554d03a52e4f508c24679a"
format_stamp: "Formatted at 2026-04-29 14:04:04 on dist-test-slave-8n77"
server_key: "1a66aab9489ea60511696a93cf6e73e5"
server_key_iv: "7d104283976a483177aeb3b35660145d"
server_key_version: "encryptionkey@0"
I20260429 14:04:04.985548 6683 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260429 14:04:04.986361 6698 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:04.986577 6683 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:04:04.986663 6683 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-1/wal
uuid: "b308843a4f554d03a52e4f508c24679a"
format_stamp: "Formatted at 2026-04-29 14:04:04 on dist-test-slave-8n77"
server_key: "1a66aab9489ea60511696a93cf6e73e5"
server_key_iv: "7d104283976a483177aeb3b35660145d"
server_key_version: "encryptionkey@0"
I20260429 14:04:04.986747 6683 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:05.003322 6683 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:05.005664 6683 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:05.005860 6683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:05.006105 6683 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:04:05.006505 6683 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:04:05.006563 6683 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:05.006597 6683 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:04:05.006615 6683 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:05.012492 6683 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.194:35449
I20260429 14:04:05.012540 6811 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.194:35449 every 8 connection(s)
I20260429 14:04:05.012850 6683 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
I20260429 14:04:05.016741 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 6683
I20260429 14:04:05.016904 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance
I20260429 14:04:05.017100 2839 external_mini_cluster.cc:1468] Setting key 304c809362b48c2f3b4340b9e54459cf
Apr 29 14:04:05 dist-test-slave-8n77 krb5kdc[6435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471444, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:04:05.022924 6814 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:05.013659 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:43919 (local address 127.2.197.194:53029)
0429 14:04:05.013890 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:05.013958 (+ 68us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:05.014569 (+ 611us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:05.014819 (+ 250us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:05.014823 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:05.015128 (+ 305us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:05.015377 (+ 249us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:05.015380 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:05.016096 (+ 716us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:05.016097 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:05.016477 (+ 380us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:05.016479 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:05.016654 (+ 175us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:05.018004 (+ 1350us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:05.018016 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:05.019502 (+ 1486us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:05.021560 (+ 2058us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:05.021568 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:05.021609 (+ 41us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:05.021846 (+ 237us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:05.022042 (+ 196us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:05.022042 (+ 0us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:05.022043 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:05.022118 (+ 75us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:05.022421 (+ 303us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:05.022425 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:05.022536 (+ 111us) client_negotiation.cc:770] Sending connection context
0429 14:04:05.022654 (+ 118us) client_negotiation.cc:241] Negotiation successful
0429 14:04:05.022766 (+ 112us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":86,"threads_started":1}
I20260429 14:04:05.022953 6523 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:05.013702 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:53029 (local address 127.2.197.254:43919)
0429 14:04:05.013975 (+ 273us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:05.013977 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:05.014059 (+ 82us) server_negotiation.cc:408] Connection header received
0429 14:04:05.014671 (+ 612us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:05.014672 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:05.014684 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:05.014746 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:05.015480 (+ 734us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:05.015989 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:05.016609 (+ 620us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:05.016791 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:05.019606 (+ 2815us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:05.019609 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:05.019610 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:05.019620 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:05.021383 (+ 1763us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:05.021934 (+ 551us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:05.021935 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:05.021936 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:05.021966 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:05.022185 (+ 219us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:05.022186 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:05.022187 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:05.022344 (+ 157us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:05.022420 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:05.022704 (+ 284us) server_negotiation.cc:300] Negotiation successful
0429 14:04:05.022813 (+ 109us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247}
I20260429 14:04:05.023501 6812 heartbeater.cc:344] Connected to a master server at 127.2.197.254:43919
I20260429 14:04:05.023624 6812 heartbeater.cc:461] Registering TS with master...
I20260429 14:04:05.023877 6812 heartbeater.cc:507] Master 127.2.197.254:43919 requested a full tablet report, sending...
I20260429 14:04:05.024299 6483 ts_manager.cc:194] Registered new tserver with Master: b308843a4f554d03a52e4f508c24679a (127.2.197.194:35449)
I20260429 14:04:05.024729 6483 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.194@KRBTEST.COM'} at 127.2.197.194:53029
WARNING: no policy specified for kudu/127.2.197.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:04:05.074904 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.195:0
--local_ip_for_outbound_sockets=127.2.197.195
--webserver_interface=127.2.197.195
--webserver_port=0
--tserver_master_addrs=127.2.197.254:43919
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:39937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:04:05.151146 6819 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:05.151392 6819 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:05.151437 6819 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:05.153144 6819 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:05.153234 6819 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:05.153281 6819 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.195
I20260429 14:04:05.155382 6819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:39937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.197.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:43919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.195
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:05.155727 6819 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:05.155944 6819 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:05.159224 6824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:05.159252 6825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:05.159456 6819 server_base.cc:1061] running on GCE node
W20260429 14:04:05.159530 6827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:05.159789 6819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:05.160059 6819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:05.161224 6819 hybrid_clock.cc:648] HybridClock initialized: now 1777471445161199 us; error 38 us; skew 500 ppm
Apr 29 14:04:05 dist-test-slave-8n77 krb5kdc[6435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471445, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:05.163995 6819 init.cc:377] Logged in from keytab as kudu/127.2.197.195@KRBTEST.COM (short username kudu)
I20260429 14:04:05.164634 6819 webserver.cc:492] Webserver started at http://127.2.197.195:34231/ using document root <none> and password file <none>
I20260429 14:04:05.164870 6819 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:05.164935 6819 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:05.165055 6819 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:04:05.166101 6819 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-2/data/instance:
uuid: "d14563ca413240678ff7b3a92c1fd1e1"
format_stamp: "Formatted at 2026-04-29 14:04:05 on dist-test-slave-8n77"
server_key: "e02869557ff10c9a0877d03718ded6a8"
server_key_iv: "004147f5bf556bfcfebaa104d4fcb2e1"
server_key_version: "encryptionkey@0"
I20260429 14:04:05.166481 6819 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance:
uuid: "d14563ca413240678ff7b3a92c1fd1e1"
format_stamp: "Formatted at 2026-04-29 14:04:05 on dist-test-slave-8n77"
server_key: "e02869557ff10c9a0877d03718ded6a8"
server_key_iv: "004147f5bf556bfcfebaa104d4fcb2e1"
server_key_version: "encryptionkey@0"
I20260429 14:04:05.167824 6819 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:04:05.168629 6834 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:05.168874 6819 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:04:05.168956 6819 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-2/wal
uuid: "d14563ca413240678ff7b3a92c1fd1e1"
format_stamp: "Formatted at 2026-04-29 14:04:05 on dist-test-slave-8n77"
server_key: "e02869557ff10c9a0877d03718ded6a8"
server_key_iv: "004147f5bf556bfcfebaa104d4fcb2e1"
server_key_version: "encryptionkey@0"
I20260429 14:04:05.169037 6819 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:05.184080 6819 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:05.186122 6819 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:05.186376 6819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:05.186641 6819 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:04:05.186998 6819 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:04:05.187057 6819 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:05.187093 6819 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:04:05.187134 6819 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:05.193516 6819 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.195:35883
I20260429 14:04:05.193584 6947 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.195:35883 every 8 connection(s)
I20260429 14:04:05.193899 6819 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
Apr 29 14:04:05 dist-test-slave-8n77 krb5kdc[6435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471445, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:04:05.199608 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 6819
I20260429 14:04:05.199697 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance
I20260429 14:04:05.199860 2839 external_mini_cluster.cc:1468] Setting key ca02437f55db26b0225dfa1d32f4fc82
I20260429 14:04:05.202903 6523 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:05.194830 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:34895 (local address 127.2.197.254:43919)
0429 14:04:05.194912 (+ 82us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:05.194915 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:05.195007 (+ 92us) server_negotiation.cc:408] Connection header received
0429 14:04:05.195451 (+ 444us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:05.195453 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:05.195462 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:05.195515 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:05.196227 (+ 712us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:05.196711 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:05.197322 (+ 611us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:05.197433 (+ 111us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:05.200063 (+ 2630us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:05.200067 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:05.200068 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:05.200075 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:05.201577 (+ 1502us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:05.201979 (+ 402us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:05.201980 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:05.201980 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:05.202010 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:05.202205 (+ 195us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:05.202206 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:05.202206 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:05.202373 (+ 167us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:05.202450 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:05.202710 (+ 260us) server_negotiation.cc:300] Negotiation successful
0429 14:04:05.202809 (+ 99us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260429 14:04:05.203034 6950 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:05.194726 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:43919 (local address 127.2.197.195:34895)
0429 14:04:05.194895 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:05.194924 (+ 29us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:05.195375 (+ 451us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:05.195588 (+ 213us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:05.195591 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:05.195896 (+ 305us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:05.196154 (+ 258us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:05.196157 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:05.196834 (+ 677us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:05.196836 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:05.197245 (+ 409us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:05.197246 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:05.197338 (+ 92us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:05.198244 (+ 906us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:05.198252 (+ 8us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:05.199848 (+ 1596us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:05.201682 (+ 1834us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:05.201687 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:05.201690 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:05.201895 (+ 205us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:05.202073 (+ 178us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:05.202074 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:05.202074 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:05.202139 (+ 65us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:05.202439 (+ 300us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:05.202444 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:05.202571 (+ 127us) client_negotiation.cc:770] Sending connection context
0429 14:04:05.202681 (+ 110us) client_negotiation.cc:241] Negotiation successful
0429 14:04:05.202823 (+ 142us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":83,"threads_started":1}
I20260429 14:04:05.203773 6948 heartbeater.cc:344] Connected to a master server at 127.2.197.254:43919
I20260429 14:04:05.203922 6948 heartbeater.cc:461] Registering TS with master...
I20260429 14:04:05.204109 6948 heartbeater.cc:507] Master 127.2.197.254:43919 requested a full tablet report, sending...
I20260429 14:04:05.204563 6483 ts_manager.cc:194] Registered new tserver with Master: d14563ca413240678ff7b3a92c1fd1e1 (127.2.197.195:35883)
I20260429 14:04:05.205027 6483 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.195@KRBTEST.COM'} at 127.2.197.195:34895
I20260429 14:04:05.212076 2839 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Apr 29 14:04:05 dist-test-slave-8n77 krb5kdc[6435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471445, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Apr 29 14:04:05 dist-test-slave-8n77 krb5kdc[6435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471445, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:04:05.234234 6523 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:05.227576 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:43206 (local address 127.2.197.254:43919)
0429 14:04:05.227650 (+ 74us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:05.227652 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:05.227853 (+ 201us) server_negotiation.cc:408] Connection header received
0429 14:04:05.227885 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:05.227886 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:05.227896 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:05.227942 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:05.228621 (+ 679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:05.229113 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:05.229770 (+ 657us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:05.229944 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:05.232131 (+ 2187us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:05.232136 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:05.232136 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:05.232143 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:05.233302 (+ 1159us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:05.233582 (+ 280us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:05.233583 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:05.233583 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:05.233601 (+ 18us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:05.233724 (+ 123us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:05.233724 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:05.233725 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:05.233843 (+ 118us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:05.233919 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:05.234071 (+ 152us) server_negotiation.cc:300] Negotiation successful
0429 14:04:05.234139 (+ 68us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260429 14:04:05.234539 6483 server_base.cc:1143] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43206
I20260429 14:04:05.235455 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 6547
I20260429 14:04:05.240559 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 6683
I20260429 14:04:05.245599 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 6819
I20260429 14:04:05.250389 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 6451
2026-04-29T14:04:05Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3019 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:04:05 dist-test-slave-8n77 krb5kdc[6960](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:04:05 dist-test-slave-8n77 krb5kdc[6960](info): set up 2 sockets
Apr 29 14:04:05 dist-test-slave-8n77 krb5kdc[6960](info): commencing operation
krb5kdc: starting...
W20260429 14:04:07.296128 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.024s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:04:07 dist-test-slave-8n77 krb5kdc[6960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:04:07Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:04:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:04:07.451836 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:43049
--webserver_interface=127.2.197.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:43415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:43049
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:04:07.530733 6976 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:07.530920 6976 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:07.530941 6976 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:07.532425 6976 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:04:07.532475 6976 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:07.532490 6976 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:07.532502 6976 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:04:07.532514 6976 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:04:07.534147 6976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:43415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:43049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:43049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:07.534443 6976 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:07.534649 6976 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:07.537472 6982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:07.537503 6981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:07.537642 6976 server_base.cc:1061] running on GCE node
W20260429 14:04:07.537467 6984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:07.537973 6976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:07.538255 6976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:07.539618 6976 hybrid_clock.cc:648] HybridClock initialized: now 1777471447539565 us; error 52 us; skew 500 ppm
Apr 29 14:04:07 dist-test-slave-8n77 krb5kdc[6960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471447, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:07.542445 6976 init.cc:377] Logged in from keytab as kudu/127.2.197.254@KRBTEST.COM (short username kudu)
I20260429 14:04:07.543035 6976 webserver.cc:492] Webserver started at http://127.2.197.254:34743/ using document root <none> and password file <none>
I20260429 14:04:07.543234 6976 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:07.543298 6976 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:07.543402 6976 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:04:07.544379 6976 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "19af5bde35c14046aef6a1cfae4fe0ca"
format_stamp: "Formatted at 2026-04-29 14:04:07 on dist-test-slave-8n77"
server_key: "c6b48ea3b7b2606ac85c2c276a7d4158"
server_key_iv: "65a9f724886c3832cd4ce635bcc0c878"
server_key_version: "encryptionkey@0"
I20260429 14:04:07.544714 6976 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "19af5bde35c14046aef6a1cfae4fe0ca"
format_stamp: "Formatted at 2026-04-29 14:04:07 on dist-test-slave-8n77"
server_key: "c6b48ea3b7b2606ac85c2c276a7d4158"
server_key_iv: "65a9f724886c3832cd4ce635bcc0c878"
server_key_version: "encryptionkey@0"
I20260429 14:04:07.545945 6976 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260429 14:04:07.546934 6991 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:07.547214 6976 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260429 14:04:07.547309 6976 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "19af5bde35c14046aef6a1cfae4fe0ca"
format_stamp: "Formatted at 2026-04-29 14:04:07 on dist-test-slave-8n77"
server_key: "c6b48ea3b7b2606ac85c2c276a7d4158"
server_key_iv: "65a9f724886c3832cd4ce635bcc0c878"
server_key_version: "encryptionkey@0"
I20260429 14:04:07.547413 6976 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:07.563405 6976 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:07.569355 6976 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:07.569531 6976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:07.573891 6976 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:43049
I20260429 14:04:07.573954 7043 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:43049 every 8 connection(s)
I20260429 14:04:07.574326 6976 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:04:07.574926 7044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:07.576694 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 6976
I20260429 14:04:07.576802 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:04:07.576963 2839 external_mini_cluster.cc:1468] Setting key ec9ea4899d984a40e276060d40576b72
I20260429 14:04:07.577555 7044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19af5bde35c14046aef6a1cfae4fe0ca: Bootstrap starting.
I20260429 14:04:07.578169 7044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19af5bde35c14046aef6a1cfae4fe0ca: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:07.578485 7044 log.cc:826] T 00000000000000000000000000000000 P 19af5bde35c14046aef6a1cfae4fe0ca: Log is configured to *not* fsync() on all Append() calls
I20260429 14:04:07.579566 7044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19af5bde35c14046aef6a1cfae4fe0ca: No bootstrap required, opened a new log
I20260429 14:04:07.581014 7044 raft_consensus.cc:359] T 00000000000000000000000000000000 P 19af5bde35c14046aef6a1cfae4fe0ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19af5bde35c14046aef6a1cfae4fe0ca" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43049 } }
I20260429 14:04:07.581127 7044 raft_consensus.cc:385] T 00000000000000000000000000000000 P 19af5bde35c14046aef6a1cfae4fe0ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:07.581147 7044 raft_consensus.cc:740] T 00000000000000000000000000000000 P 19af5bde35c14046aef6a1cfae4fe0ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19af5bde35c14046aef6a1cfae4fe0ca, State: Initialized, Role: FOLLOWER
I20260429 14:04:07.581323 7044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19af5bde35c14046aef6a1cfae4fe0ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19af5bde35c14046aef6a1cfae4fe0ca" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43049 } }
I20260429 14:04:07.581406 7044 raft_consensus.cc:399] T 00000000000000000000000000000000 P 19af5bde35c14046aef6a1cfae4fe0ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:04:07.581440 7044 raft_consensus.cc:493] T 00000000000000000000000000000000 P 19af5bde35c14046aef6a1cfae4fe0ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:04:07.581480 7044 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 19af5bde35c14046aef6a1cfae4fe0ca [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:07.582345 7044 raft_consensus.cc:515] T 00000000000000000000000000000000 P 19af5bde35c14046aef6a1cfae4fe0ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19af5bde35c14046aef6a1cfae4fe0ca" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43049 } }
I20260429 14:04:07.582473 7044 leader_election.cc:304] T 00000000000000000000000000000000 P 19af5bde35c14046aef6a1cfae4fe0ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19af5bde35c14046aef6a1cfae4fe0ca; no voters:
I20260429 14:04:07.582667 7044 leader_election.cc:290] T 00000000000000000000000000000000 P 19af5bde35c14046aef6a1cfae4fe0ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:04:07.582836 7049 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 19af5bde35c14046aef6a1cfae4fe0ca [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:04:07.582914 7044 sys_catalog.cc:565] T 00000000000000000000000000000000 P 19af5bde35c14046aef6a1cfae4fe0ca [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:04:07.583067 7049 raft_consensus.cc:697] T 00000000000000000000000000000000 P 19af5bde35c14046aef6a1cfae4fe0ca [term 1 LEADER]: Becoming Leader. State: Replica: 19af5bde35c14046aef6a1cfae4fe0ca, State: Running, Role: LEADER
I20260429 14:04:07.583262 7049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19af5bde35c14046aef6a1cfae4fe0ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19af5bde35c14046aef6a1cfae4fe0ca" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43049 } }
Apr 29 14:04:07 dist-test-slave-8n77 krb5kdc[6960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:04:07.583635 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19af5bde35c14046aef6a1cfae4fe0ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19af5bde35c14046aef6a1cfae4fe0ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19af5bde35c14046aef6a1cfae4fe0ca" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43049 } } }
I20260429 14:04:07.583727 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19af5bde35c14046aef6a1cfae4fe0ca [sys.catalog]: This master's current role is: LEADER
I20260429 14:04:07.584172 7051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19af5bde35c14046aef6a1cfae4fe0ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19af5bde35c14046aef6a1cfae4fe0ca. Latest consensus state: current_term: 1 leader_uuid: "19af5bde35c14046aef6a1cfae4fe0ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19af5bde35c14046aef6a1cfae4fe0ca" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43049 } } }
I20260429 14:04:07.584308 7051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19af5bde35c14046aef6a1cfae4fe0ca [sys.catalog]: This master's current role is: LEADER
W20260429 14:04:07.584941 7065 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 19af5bde35c14046aef6a1cfae4fe0ca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260429 14:04:07.585194 7065 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260429 14:04:07.585282 7058 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:04:07.585481 7058 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:04:07.586911 7058 catalog_manager.cc:1357] Generated new cluster ID: dfd850646a4f43b19f4f8e9190ded756
I20260429 14:04:07.586968 7058 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:04:07.587854 7047 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:07.577563 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54770 (local address 127.2.197.254:43049)
0429 14:04:07.577771 (+ 208us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:07.577780 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:07.577798 (+ 18us) server_negotiation.cc:408] Connection header received
0429 14:04:07.578108 (+ 310us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:07.578114 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:07.578150 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:07.578328 (+ 178us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:07.579151 (+ 823us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:07.579998 (+ 847us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:07.580764 (+ 766us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:07.581000 (+ 236us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:07.584070 (+ 3070us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:07.584079 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:07.584079 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:07.584093 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:07.586243 (+ 2150us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:07.586691 (+ 448us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:07.586694 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:07.586695 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:07.586748 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:07.587042 (+ 294us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:07.587043 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:07.587044 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:07.587274 (+ 230us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:07.587437 (+ 163us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:07.587550 (+ 113us) server_negotiation.cc:300] Negotiation successful
0429 14:04:07.587685 (+ 135us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":99,"threads_started":1}
I20260429 14:04:07.592419 7058 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:04:07.593161 7058 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:04:07.609217 7058 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 19af5bde35c14046aef6a1cfae4fe0ca: Generated new TSK 0
I20260429 14:04:07.609417 7058 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:04:07.670131 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.193:0
--local_ip_for_outbound_sockets=127.2.197.193
--webserver_interface=127.2.197.193
--webserver_port=0
--tserver_master_addrs=127.2.197.254:43049
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:43415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:04:07.752417 7072 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:07.752595 7072 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:07.752620 7072 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:07.754247 7072 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:07.754369 7072 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:07.754424 7072 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.193
I20260429 14:04:07.756150 7072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:43415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.197.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:43049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.193
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:07.756398 7072 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:07.756625 7072 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:07.759539 7078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:07.759557 7077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:07.759539 7080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:07.759945 7072 server_base.cc:1061] running on GCE node
I20260429 14:04:07.760151 7072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:07.760367 7072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:07.761502 7072 hybrid_clock.cc:648] HybridClock initialized: now 1777471447761497 us; error 74 us; skew 500 ppm
Apr 29 14:04:07 dist-test-slave-8n77 krb5kdc[6960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471447, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:07.764953 7072 init.cc:377] Logged in from keytab as kudu/127.2.197.193@KRBTEST.COM (short username kudu)
I20260429 14:04:07.765602 7072 webserver.cc:492] Webserver started at http://127.2.197.193:46323/ using document root <none> and password file <none>
I20260429 14:04:07.765817 7072 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:07.765880 7072 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:07.765991 7072 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:04:07.767117 7072 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-0/data/instance:
uuid: "b8e4559ad2b04ea591f5ed7e50256510"
format_stamp: "Formatted at 2026-04-29 14:04:07 on dist-test-slave-8n77"
server_key: "d55063f994eb32a6dafffadc7e0d2865"
server_key_iv: "faa7dbb6fb189aa1e1ba4a97ef393024"
server_key_version: "encryptionkey@0"
I20260429 14:04:07.767472 7072 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance:
uuid: "b8e4559ad2b04ea591f5ed7e50256510"
format_stamp: "Formatted at 2026-04-29 14:04:07 on dist-test-slave-8n77"
server_key: "d55063f994eb32a6dafffadc7e0d2865"
server_key_iv: "faa7dbb6fb189aa1e1ba4a97ef393024"
server_key_version: "encryptionkey@0"
I20260429 14:04:07.768813 7072 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260429 14:04:07.769613 7087 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:07.769831 7072 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:07.769927 7072 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-0/wal
uuid: "b8e4559ad2b04ea591f5ed7e50256510"
format_stamp: "Formatted at 2026-04-29 14:04:07 on dist-test-slave-8n77"
server_key: "d55063f994eb32a6dafffadc7e0d2865"
server_key_iv: "faa7dbb6fb189aa1e1ba4a97ef393024"
server_key_version: "encryptionkey@0"
I20260429 14:04:07.770016 7072 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:07.803130 7072 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:07.805733 7072 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:07.805949 7072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:07.806267 7072 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:04:07.806914 7072 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:04:07.806988 7072 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:07.807039 7072 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:04:07.807065 7072 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:07.814994 7072 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.193:41757
I20260429 14:04:07.815106 7200 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.193:41757 every 8 connection(s)
I20260429 14:04:07.815418 7072 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
I20260429 14:04:07.815960 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 7072
I20260429 14:04:07.816069 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance
I20260429 14:04:07.816239 2839 external_mini_cluster.cc:1468] Setting key ff7a49d3bec1188cf0d5d0f65427024f
Apr 29 14:04:07 dist-test-slave-8n77 krb5kdc[6960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471447, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.197.194@KRBTEST.COM; defaulting to no policy
I20260429 14:04:07.826556 7047 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:07.816466 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:33711 (local address 127.2.197.254:43049)
0429 14:04:07.816537 (+ 71us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:07.816539 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:07.816935 (+ 396us) server_negotiation.cc:408] Connection header received
0429 14:04:07.817420 (+ 485us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:07.817421 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:07.817434 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:07.817509 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:07.818266 (+ 757us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:07.818814 (+ 548us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:07.819630 (+ 816us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:07.819783 (+ 153us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:07.822668 (+ 2885us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:07.822673 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:07.822673 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:07.822681 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:07.824823 (+ 2142us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:07.825405 (+ 582us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:07.825406 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:07.825407 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:07.825431 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:07.825714 (+ 283us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:07.825715 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:07.825715 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:07.825857 (+ 142us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:07.825927 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:07.826388 (+ 461us) server_negotiation.cc:300] Negotiation successful
0429 14:04:07.826485 (+ 97us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260429 14:04:07.826624 7203 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:07.816597 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:43049 (local address 127.2.197.193:33711)
0429 14:04:07.816791 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:07.816828 (+ 37us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:07.817311 (+ 483us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:07.817560 (+ 249us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:07.817571 (+ 11us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:07.817891 (+ 320us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:07.818150 (+ 259us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:07.818153 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:07.818920 (+ 767us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:07.818921 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:07.819525 (+ 604us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:07.819528 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:07.819653 (+ 125us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:07.820767 (+ 1114us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:07.820779 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:07.822549 (+ 1770us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:07.824994 (+ 2445us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:07.825003 (+ 9us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:07.825007 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:07.825307 (+ 300us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:07.825533 (+ 226us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:07.825534 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:07.825534 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:07.825610 (+ 76us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:07.825952 (+ 342us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:07.825957 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:07.826178 (+ 221us) client_negotiation.cc:770] Sending connection context
0429 14:04:07.826327 (+ 149us) client_negotiation.cc:241] Negotiation successful
0429 14:04:07.826470 (+ 143us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":112,"threads_started":1}
I20260429 14:04:07.827248 7201 heartbeater.cc:344] Connected to a master server at 127.2.197.254:43049
I20260429 14:04:07.827354 7201 heartbeater.cc:461] Registering TS with master...
I20260429 14:04:07.827626 7201 heartbeater.cc:507] Master 127.2.197.254:43049 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.194@KRBTEST.COM" created.
I20260429 14:04:07.828287 7008 ts_manager.cc:194] Registered new tserver with Master: b8e4559ad2b04ea591f5ed7e50256510 (127.2.197.193:41757)
I20260429 14:04:07.828962 7008 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.193@KRBTEST.COM'} at 127.2.197.193:33711
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:04:07.878526 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.194:0
--local_ip_for_outbound_sockets=127.2.197.194
--webserver_interface=127.2.197.194
--webserver_port=0
--tserver_master_addrs=127.2.197.254:43049
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:43415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:04:07.961174 7208 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:07.961360 7208 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:07.961385 7208 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:07.963053 7208 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:07.963110 7208 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:07.963140 7208 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.194
I20260429 14:04:07.964989 7208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:43415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.197.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:43049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.194
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:07.965243 7208 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:07.965453 7208 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:07.968462 7213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:07.968515 7216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:07.968588 7208 server_base.cc:1061] running on GCE node
W20260429 14:04:07.968595 7214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:07.968987 7208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:07.969228 7208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:07.970389 7208 hybrid_clock.cc:648] HybridClock initialized: now 1777471447970382 us; error 39 us; skew 500 ppm
Apr 29 14:04:07 dist-test-slave-8n77 krb5kdc[6960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471447, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:07.973004 7208 init.cc:377] Logged in from keytab as kudu/127.2.197.194@KRBTEST.COM (short username kudu)
I20260429 14:04:07.973642 7208 webserver.cc:492] Webserver started at http://127.2.197.194:37153/ using document root <none> and password file <none>
I20260429 14:04:07.973856 7208 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:07.973898 7208 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:07.974010 7208 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:04:07.975037 7208 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-1/data/instance:
uuid: "a318bca8dd384744a46247c4bc01cffb"
format_stamp: "Formatted at 2026-04-29 14:04:07 on dist-test-slave-8n77"
server_key: "9c6d89954812dd5f03d2318c0853ffa8"
server_key_iv: "034396774bbaefe51a02c4b4dc5b1b36"
server_key_version: "encryptionkey@0"
I20260429 14:04:07.975366 7208 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance:
uuid: "a318bca8dd384744a46247c4bc01cffb"
format_stamp: "Formatted at 2026-04-29 14:04:07 on dist-test-slave-8n77"
server_key: "9c6d89954812dd5f03d2318c0853ffa8"
server_key_iv: "034396774bbaefe51a02c4b4dc5b1b36"
server_key_version: "encryptionkey@0"
I20260429 14:04:07.977133 7208 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260429 14:04:07.978597 7223 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:07.978909 7208 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:07.979000 7208 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-1/wal
uuid: "a318bca8dd384744a46247c4bc01cffb"
format_stamp: "Formatted at 2026-04-29 14:04:07 on dist-test-slave-8n77"
server_key: "9c6d89954812dd5f03d2318c0853ffa8"
server_key_iv: "034396774bbaefe51a02c4b4dc5b1b36"
server_key_version: "encryptionkey@0"
I20260429 14:04:07.979113 7208 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:07.995144 7208 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:07.998204 7208 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:07.998420 7208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:07.998683 7208 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:04:07.999025 7208 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:04:07.999080 7208 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:07.999158 7208 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:04:07.999190 7208 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:08.006959 7208 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.194:37685
I20260429 14:04:08.007045 7336 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.194:37685 every 8 connection(s)
I20260429 14:04:08.007362 7208 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
Apr 29 14:04:08 dist-test-slave-8n77 krb5kdc[6960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471447, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:04:08.014396 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 7208
I20260429 14:04:08.014568 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance
I20260429 14:04:08.014770 2839 external_mini_cluster.cc:1468] Setting key b647a3bf6238f77529f81ba62279d582
I20260429 14:04:08.017601 7047 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:08.008283 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:33611 (local address 127.2.197.254:43049)
0429 14:04:08.008364 (+ 81us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:08.008366 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:08.008536 (+ 170us) server_negotiation.cc:408] Connection header received
0429 14:04:08.009006 (+ 470us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:08.009007 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:08.009017 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:08.009081 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:08.009863 (+ 782us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:08.010440 (+ 577us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:08.011177 (+ 737us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:08.011523 (+ 346us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:08.013860 (+ 2337us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:08.013867 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:08.013868 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:08.013877 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:08.015841 (+ 1964us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:08.016379 (+ 538us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:08.016381 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:08.016382 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:08.016410 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:08.016712 (+ 302us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:08.016713 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:08.016713 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:08.016859 (+ 146us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:08.016932 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:08.017381 (+ 449us) server_negotiation.cc:300] Negotiation successful
0429 14:04:08.017497 (+ 116us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260429 14:04:08.017764 7339 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:08.008233 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:43049 (local address 127.2.197.194:33611)
0429 14:04:08.008430 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:08.008462 (+ 32us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:08.008929 (+ 467us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:08.009151 (+ 222us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:08.009155 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:08.009454 (+ 299us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:08.009750 (+ 296us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:08.009753 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:08.010593 (+ 840us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:08.010594 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:08.010983 (+ 389us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:08.010985 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:08.011097 (+ 112us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:08.012226 (+ 1129us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:08.012235 (+ 9us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:08.013703 (+ 1468us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:08.015953 (+ 2250us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:08.015965 (+ 12us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:08.015968 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:08.016284 (+ 316us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:08.016533 (+ 249us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:08.016535 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:08.016535 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:08.016634 (+ 99us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:08.016949 (+ 315us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:08.016961 (+ 12us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:08.017202 (+ 241us) client_negotiation.cc:770] Sending connection context
0429 14:04:08.017356 (+ 154us) client_negotiation.cc:241] Negotiation successful
0429 14:04:08.017537 (+ 181us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":86,"threads_started":1}
I20260429 14:04:08.018312 7337 heartbeater.cc:344] Connected to a master server at 127.2.197.254:43049
I20260429 14:04:08.018419 7337 heartbeater.cc:461] Registering TS with master...
I20260429 14:04:08.018635 7337 heartbeater.cc:507] Master 127.2.197.254:43049 requested a full tablet report, sending...
I20260429 14:04:08.019172 7008 ts_manager.cc:194] Registered new tserver with Master: a318bca8dd384744a46247c4bc01cffb (127.2.197.194:37685)
I20260429 14:04:08.019740 7008 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.194@KRBTEST.COM'} at 127.2.197.194:33611
WARNING: no policy specified for kudu/127.2.197.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:04:08.078614 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.195:0
--local_ip_for_outbound_sockets=127.2.197.195
--webserver_interface=127.2.197.195
--webserver_port=0
--tserver_master_addrs=127.2.197.254:43049
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:43415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:04:08.158247 7344 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:08.158502 7344 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:08.158546 7344 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:08.160136 7344 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:08.160226 7344 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:08.160274 7344 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.195
I20260429 14:04:08.161993 7344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:43415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.197.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:43049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.195
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:08.162264 7344 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:08.162489 7344 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:08.165266 7352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:08.165210 7349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:08.165237 7350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:08.165688 7344 server_base.cc:1061] running on GCE node
I20260429 14:04:08.165881 7344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:08.166131 7344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:08.167299 7344 hybrid_clock.cc:648] HybridClock initialized: now 1777471448167286 us; error 35 us; skew 500 ppm
Apr 29 14:04:08 dist-test-slave-8n77 krb5kdc[6960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471448, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:08.169941 7344 init.cc:377] Logged in from keytab as kudu/127.2.197.195@KRBTEST.COM (short username kudu)
I20260429 14:04:08.170560 7344 webserver.cc:492] Webserver started at http://127.2.197.195:43727/ using document root <none> and password file <none>
I20260429 14:04:08.170760 7344 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:08.170819 7344 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:08.170922 7344 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:04:08.172008 7344 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-2/data/instance:
uuid: "382640e52a8842deab7f659c83af577a"
format_stamp: "Formatted at 2026-04-29 14:04:08 on dist-test-slave-8n77"
server_key: "7997ef166c50e5fa1bed544d46e8893e"
server_key_iv: "04f8d85457916cf990de4b44c3e5bafa"
server_key_version: "encryptionkey@0"
I20260429 14:04:08.172361 7344 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance:
uuid: "382640e52a8842deab7f659c83af577a"
format_stamp: "Formatted at 2026-04-29 14:04:08 on dist-test-slave-8n77"
server_key: "7997ef166c50e5fa1bed544d46e8893e"
server_key_iv: "04f8d85457916cf990de4b44c3e5bafa"
server_key_version: "encryptionkey@0"
I20260429 14:04:08.173981 7344 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260429 14:04:08.174778 7359 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:08.175120 7344 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260429 14:04:08.175238 7344 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-2/wal
uuid: "382640e52a8842deab7f659c83af577a"
format_stamp: "Formatted at 2026-04-29 14:04:08 on dist-test-slave-8n77"
server_key: "7997ef166c50e5fa1bed544d46e8893e"
server_key_iv: "04f8d85457916cf990de4b44c3e5bafa"
server_key_version: "encryptionkey@0"
I20260429 14:04:08.175324 7344 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:08.190619 7344 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:08.193092 7344 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:08.193284 7344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:08.193569 7344 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:04:08.193958 7344 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:04:08.194018 7344 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:08.194064 7344 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:04:08.194094 7344 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:08.200605 7344 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.195:39157
I20260429 14:04:08.200644 7472 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.195:39157 every 8 connection(s)
I20260429 14:04:08.200984 7344 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
I20260429 14:04:08.203413 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 7344
I20260429 14:04:08.203495 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance
I20260429 14:04:08.203645 2839 external_mini_cluster.cc:1468] Setting key 53bdc53c467acfd031c77e676cc2a314
Apr 29 14:04:08 dist-test-slave-8n77 krb5kdc[6960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471448, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:04:08.209513 7047 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:08.201759 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:34189 (local address 127.2.197.254:43049)
0429 14:04:08.201863 (+ 104us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:08.201866 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:08.201968 (+ 102us) server_negotiation.cc:408] Connection header received
0429 14:04:08.202463 (+ 495us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:08.202464 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:08.202474 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:08.202527 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:08.203250 (+ 723us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:08.203987 (+ 737us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:08.204727 (+ 740us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:08.204863 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:08.207221 (+ 2358us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:08.207225 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:08.207225 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:08.207232 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:08.208455 (+ 1223us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:08.208801 (+ 346us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:08.208802 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:08.208802 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:08.208828 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:08.208989 (+ 161us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:08.208989 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:08.208989 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:08.209069 (+ 80us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:08.209118 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:08.209376 (+ 258us) server_negotiation.cc:300] Negotiation successful
0429 14:04:08.209446 (+ 70us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260429 14:04:08.209712 7475 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:08.201712 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:43049 (local address 127.2.197.195:34189)
0429 14:04:08.201861 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:08.201890 (+ 29us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:08.202385 (+ 495us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:08.202568 (+ 183us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:08.202571 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:08.202882 (+ 311us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:08.203144 (+ 262us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:08.203148 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:08.204098 (+ 950us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:08.204100 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:08.204652 (+ 552us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:08.204654 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:08.204753 (+ 99us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:08.205673 (+ 920us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:08.205682 (+ 9us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:08.207144 (+ 1462us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:08.208531 (+ 1387us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:08.208536 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:08.208538 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:08.208737 (+ 199us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:08.208887 (+ 150us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:08.208888 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:08.208888 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:08.208940 (+ 52us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:08.209146 (+ 206us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:08.209150 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:08.209259 (+ 109us) client_negotiation.cc:770] Sending connection context
0429 14:04:08.209358 (+ 99us) client_negotiation.cc:241] Negotiation successful
0429 14:04:08.209463 (+ 105us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
I20260429 14:04:08.210269 7473 heartbeater.cc:344] Connected to a master server at 127.2.197.254:43049
I20260429 14:04:08.210389 7473 heartbeater.cc:461] Registering TS with master...
I20260429 14:04:08.210536 7473 heartbeater.cc:507] Master 127.2.197.254:43049 requested a full tablet report, sending...
I20260429 14:04:08.210901 7008 ts_manager.cc:194] Registered new tserver with Master: 382640e52a8842deab7f659c83af577a (127.2.197.195:39157)
I20260429 14:04:08.211246 7008 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.195@KRBTEST.COM'} at 127.2.197.195:34189
I20260429 14:04:08.215934 2839 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Apr 29 14:04:08 dist-test-slave-8n77 krb5kdc[6960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Apr 29 14:04:08 dist-test-slave-8n77 krb5kdc[6960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.193@KRBTEST.COM
I20260429 14:04:08.240178 7480 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:08.231291 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48754 (local address 127.2.197.193:41757)
0429 14:04:08.231481 (+ 190us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:08.231490 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:08.231502 (+ 12us) server_negotiation.cc:408] Connection header received
0429 14:04:08.231624 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:08.231628 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:08.231645 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:08.231774 (+ 129us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:08.232536 (+ 762us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:08.233245 (+ 709us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:08.234123 (+ 878us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:08.234529 (+ 406us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:08.236941 (+ 2412us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:08.236986 (+ 45us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:08.236987 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:08.236995 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:08.238984 (+ 1989us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:08.239374 (+ 390us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:08.239381 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:08.239381 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:08.239406 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:08.239595 (+ 189us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:08.239596 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:08.239597 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:08.239716 (+ 119us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:08.239864 (+ 148us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:08.239949 (+ 85us) server_negotiation.cc:300] Negotiation successful
0429 14:04:08.240056 (+ 107us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":77,"threads_started":1}
Apr 29 14:04:08 dist-test-slave-8n77 krb5kdc[6960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:04:08.249032 7047 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:08.241682 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36752 (local address 127.2.197.254:43049)
0429 14:04:08.241762 (+ 80us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:08.241764 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:08.241821 (+ 57us) server_negotiation.cc:408] Connection header received
0429 14:04:08.241865 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:08.241866 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:08.241876 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:08.241948 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:08.242765 (+ 817us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:08.243320 (+ 555us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:08.244159 (+ 839us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:08.244326 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:08.246637 (+ 2311us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:08.246641 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:08.246641 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:08.246649 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:08.247929 (+ 1280us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:08.248258 (+ 329us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:08.248259 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:08.248260 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:08.248284 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:08.248496 (+ 212us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:08.248497 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:08.248498 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:08.248639 (+ 141us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:08.248733 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:08.248846 (+ 113us) server_negotiation.cc:300] Negotiation successful
0429 14:04:08.248918 (+ 72us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260429 14:04:08.249325 7008 server_base.cc:1143] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36752
I20260429 14:04:08.249750 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 7072
I20260429 14:04:08.255031 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 7208
I20260429 14:04:08.260610 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 7344
I20260429 14:04:08.265422 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 6976
2026-04-29T14:04:08Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3014 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:04:08 dist-test-slave-8n77 krb5kdc[7485](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:04:08 dist-test-slave-8n77 krb5kdc[7485](info): set up 2 sockets
Apr 29 14:04:08 dist-test-slave-8n77 krb5kdc[7485](info): commencing operation
krb5kdc: starting...
W20260429 14:04:10.307924 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.021s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:04:10 dist-test-slave-8n77 krb5kdc[7485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:04:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:04:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:04:10.462692 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:43453
--webserver_interface=127.2.197.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:35141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:43453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:04:10.542577 7501 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:10.542842 7501 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:10.542889 7501 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:10.544456 7501 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:04:10.544559 7501 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:10.544581 7501 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:10.544605 7501 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:04:10.544633 7501 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:04:10.546428 7501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:35141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:43453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:43453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:10.546751 7501 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:10.546957 7501 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260429 14:04:10.549427 7501 server_base.cc:1061] running on GCE node
W20260429 14:04:10.550019 7509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:10.550019 7506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:10.550086 7507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:10.550853 7501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:10.551120 7501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:10.552305 7501 hybrid_clock.cc:648] HybridClock initialized: now 1777471450552285 us; error 43 us; skew 500 ppm
Apr 29 14:04:10 dist-test-slave-8n77 krb5kdc[7485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471450, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:10.554836 7501 init.cc:377] Logged in from keytab as kudu/127.2.197.254@KRBTEST.COM (short username kudu)
I20260429 14:04:10.555188 7501 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:10.555263 7501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:10.555398 7501 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:04:10.556408 7501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "4206d9b586cf425e9c6afe0dfd3483c2"
format_stamp: "Formatted at 2026-04-29 14:04:10 on dist-test-slave-8n77"
server_key: "978bf55bf0254cb7f75d86282d1520b9"
server_key_iv: "755c28eeec470836bb4493844e4529e3"
server_key_version: "encryptionkey@0"
I20260429 14:04:10.556748 7501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "4206d9b586cf425e9c6afe0dfd3483c2"
format_stamp: "Formatted at 2026-04-29 14:04:10 on dist-test-slave-8n77"
server_key: "978bf55bf0254cb7f75d86282d1520b9"
server_key_iv: "755c28eeec470836bb4493844e4529e3"
server_key_version: "encryptionkey@0"
I20260429 14:04:10.558058 7501 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260429 14:04:10.558954 7515 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:10.559160 7501 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:10.559250 7501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "4206d9b586cf425e9c6afe0dfd3483c2"
format_stamp: "Formatted at 2026-04-29 14:04:10 on dist-test-slave-8n77"
server_key: "978bf55bf0254cb7f75d86282d1520b9"
server_key_iv: "755c28eeec470836bb4493844e4529e3"
server_key_version: "encryptionkey@0"
I20260429 14:04:10.559337 7501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:10.584762 7501 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:10.586896 7501 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:10.587105 7501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:10.591655 7501 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:43453
I20260429 14:04:10.591759 7567 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:43453 every 8 connection(s)
I20260429 14:04:10.592060 7501 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:04:10.592746 7568 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:10.595357 7568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4206d9b586cf425e9c6afe0dfd3483c2: Bootstrap starting.
I20260429 14:04:10.596002 7568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4206d9b586cf425e9c6afe0dfd3483c2: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:10.596278 7568 log.cc:826] T 00000000000000000000000000000000 P 4206d9b586cf425e9c6afe0dfd3483c2: Log is configured to *not* fsync() on all Append() calls
I20260429 14:04:10.597048 7568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4206d9b586cf425e9c6afe0dfd3483c2: No bootstrap required, opened a new log
I20260429 14:04:10.598260 7568 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4206d9b586cf425e9c6afe0dfd3483c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4206d9b586cf425e9c6afe0dfd3483c2" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43453 } }
I20260429 14:04:10.598464 7568 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4206d9b586cf425e9c6afe0dfd3483c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:10.598486 7568 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4206d9b586cf425e9c6afe0dfd3483c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4206d9b586cf425e9c6afe0dfd3483c2, State: Initialized, Role: FOLLOWER
I20260429 14:04:10.598702 7568 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4206d9b586cf425e9c6afe0dfd3483c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4206d9b586cf425e9c6afe0dfd3483c2" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43453 } }
I20260429 14:04:10.598786 7568 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4206d9b586cf425e9c6afe0dfd3483c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:04:10.598821 7568 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4206d9b586cf425e9c6afe0dfd3483c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:04:10.598873 7568 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4206d9b586cf425e9c6afe0dfd3483c2 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:10.598908 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 7501
I20260429 14:04:10.599043 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:04:10.599228 2839 external_mini_cluster.cc:1468] Setting key bda1df71da0f669ddd77ac02073f0a93
I20260429 14:04:10.599581 7568 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4206d9b586cf425e9c6afe0dfd3483c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4206d9b586cf425e9c6afe0dfd3483c2" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43453 } }
I20260429 14:04:10.599705 7568 leader_election.cc:304] T 00000000000000000000000000000000 P 4206d9b586cf425e9c6afe0dfd3483c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4206d9b586cf425e9c6afe0dfd3483c2; no voters:
I20260429 14:04:10.599908 7568 leader_election.cc:290] T 00000000000000000000000000000000 P 4206d9b586cf425e9c6afe0dfd3483c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:04:10.599944 7573 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4206d9b586cf425e9c6afe0dfd3483c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:04:10.600046 7573 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4206d9b586cf425e9c6afe0dfd3483c2 [term 1 LEADER]: Becoming Leader. State: Replica: 4206d9b586cf425e9c6afe0dfd3483c2, State: Running, Role: LEADER
I20260429 14:04:10.600121 7573 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4206d9b586cf425e9c6afe0dfd3483c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4206d9b586cf425e9c6afe0dfd3483c2" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43453 } }
I20260429 14:04:10.600148 7568 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4206d9b586cf425e9c6afe0dfd3483c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:04:10.600555 7575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4206d9b586cf425e9c6afe0dfd3483c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4206d9b586cf425e9c6afe0dfd3483c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4206d9b586cf425e9c6afe0dfd3483c2" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43453 } } }
I20260429 14:04:10.600546 7574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4206d9b586cf425e9c6afe0dfd3483c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4206d9b586cf425e9c6afe0dfd3483c2. Latest consensus state: current_term: 1 leader_uuid: "4206d9b586cf425e9c6afe0dfd3483c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4206d9b586cf425e9c6afe0dfd3483c2" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43453 } } }
I20260429 14:04:10.600757 7574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4206d9b586cf425e9c6afe0dfd3483c2 [sys.catalog]: This master's current role is: LEADER
I20260429 14:04:10.600968 7575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4206d9b586cf425e9c6afe0dfd3483c2 [sys.catalog]: This master's current role is: LEADER
I20260429 14:04:10.601449 7578 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:04:10.602078 7578 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:04:10.604063 7578 catalog_manager.cc:1357] Generated new cluster ID: 99d373e56ece43d79d4617bf86da87cb
I20260429 14:04:10.604127 7578 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Apr 29 14:04:10 dist-test-slave-8n77 krb5kdc[7485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:04:10.610663 7572 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:10.599728 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60032 (local address 127.2.197.254:43453)
0429 14:04:10.599876 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:10.599888 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:10.600072 (+ 184us) server_negotiation.cc:408] Connection header received
0429 14:04:10.600285 (+ 213us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:10.600291 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:10.600343 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:10.600452 (+ 109us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:10.601555 (+ 1103us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:10.602646 (+ 1091us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:10.603480 (+ 834us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:10.603736 (+ 256us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:10.606238 (+ 2502us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:10.606245 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:10.606246 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:10.606261 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:10.608355 (+ 2094us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:10.608751 (+ 396us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:10.608754 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:10.608754 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:10.608992 (+ 238us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:10.609257 (+ 265us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:10.609258 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:10.609258 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:10.609543 (+ 285us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:10.609729 (+ 186us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:10.610306 (+ 577us) server_negotiation.cc:300] Negotiation successful
0429 14:04:10.610449 (+ 143us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":71,"threads_started":1}
I20260429 14:04:10.621121 7578 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:04:10.621696 7578 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:04:10.629997 7578 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 4206d9b586cf425e9c6afe0dfd3483c2: Generated new TSK 0
I20260429 14:04:10.630218 7578 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:04:10.694394 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.193:0
--local_ip_for_outbound_sockets=127.2.197.193
--webserver_interface=127.2.197.193
--webserver_port=0
--tserver_master_addrs=127.2.197.254:43453
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:35141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:04:10.776573 7596 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:10.776799 7596 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:10.776839 7596 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:10.778419 7596 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:10.778487 7596 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:10.778533 7596 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.193
I20260429 14:04:10.780340 7596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:35141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.197.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:43453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.193
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:10.780632 7596 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:10.780830 7596 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:10.783664 7601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:10.783789 7596 server_base.cc:1061] running on GCE node
W20260429 14:04:10.783664 7604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:10.783699 7602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:10.784073 7596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:10.784317 7596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:10.785482 7596 hybrid_clock.cc:648] HybridClock initialized: now 1777471450785435 us; error 57 us; skew 500 ppm
Apr 29 14:04:10 dist-test-slave-8n77 krb5kdc[7485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471450, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:10.788445 7596 init.cc:377] Logged in from keytab as kudu/127.2.197.193@KRBTEST.COM (short username kudu)
I20260429 14:04:10.788815 7596 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:10.788877 7596 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:10.788983 7596 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:04:10.789974 7596 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-0/data/instance:
uuid: "5ea270dcbade4481a05fb51e3ac91671"
format_stamp: "Formatted at 2026-04-29 14:04:10 on dist-test-slave-8n77"
server_key: "fe8585e564aa58f461aa36aa959dd0b5"
server_key_iv: "ff9d8ff5bc1cea37cd4f6fd484202070"
server_key_version: "encryptionkey@0"
I20260429 14:04:10.790369 7596 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance:
uuid: "5ea270dcbade4481a05fb51e3ac91671"
format_stamp: "Formatted at 2026-04-29 14:04:10 on dist-test-slave-8n77"
server_key: "fe8585e564aa58f461aa36aa959dd0b5"
server_key_iv: "ff9d8ff5bc1cea37cd4f6fd484202070"
server_key_version: "encryptionkey@0"
I20260429 14:04:10.791602 7596 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:10.792438 7610 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:10.792855 7596 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:10.792923 7596 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-0/wal
uuid: "5ea270dcbade4481a05fb51e3ac91671"
format_stamp: "Formatted at 2026-04-29 14:04:10 on dist-test-slave-8n77"
server_key: "fe8585e564aa58f461aa36aa959dd0b5"
server_key_iv: "ff9d8ff5bc1cea37cd4f6fd484202070"
server_key_version: "encryptionkey@0"
I20260429 14:04:10.793026 7596 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:10.818768 7596 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:10.820814 7596 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:10.820916 7596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:10.821198 7596 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:04:10.821527 7596 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:04:10.821560 7596 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:10.821617 7596 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:04:10.821652 7596 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:10.828089 7596 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.193:33607
I20260429 14:04:10.828130 7723 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.193:33607 every 8 connection(s)
I20260429 14:04:10.828388 7596 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
I20260429 14:04:10.829814 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 7596
I20260429 14:04:10.829939 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance
I20260429 14:04:10.830124 2839 external_mini_cluster.cc:1468] Setting key d4afafcf4e8072de4b801c80bfb7fa9f
Apr 29 14:04:10 dist-test-slave-8n77 krb5kdc[7485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471450, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.197.194@KRBTEST.COM; defaulting to no policy
I20260429 14:04:10.837618 7572 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:10.829139 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:58573 (local address 127.2.197.254:43453)
0429 14:04:10.829220 (+ 81us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:10.829223 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:10.829378 (+ 155us) server_negotiation.cc:408] Connection header received
0429 14:04:10.830077 (+ 699us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:10.830079 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:10.830096 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:10.830153 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:10.831480 (+ 1327us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:10.831989 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:10.832663 (+ 674us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:10.832850 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:10.834919 (+ 2069us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:10.834923 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:10.834924 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:10.834932 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:10.836321 (+ 1389us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:10.836735 (+ 414us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:10.836736 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:10.836737 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:10.836768 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:10.836995 (+ 227us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:10.836996 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:10.836997 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:10.837100 (+ 103us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:10.837176 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:10.837425 (+ 249us) server_negotiation.cc:300] Negotiation successful
0429 14:04:10.837520 (+ 95us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260429 14:04:10.837740 7726 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:10.829107 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:43453 (local address 127.2.197.193:58573)
0429 14:04:10.829270 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:10.829298 (+ 28us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:10.829902 (+ 604us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:10.830219 (+ 317us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:10.830224 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:10.830616 (+ 392us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:10.831380 (+ 764us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:10.831385 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:10.832083 (+ 698us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:10.832084 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:10.832565 (+ 481us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:10.832566 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:10.832707 (+ 141us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:10.833326 (+ 619us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:10.833335 (+ 9us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:10.834821 (+ 1486us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:10.836432 (+ 1611us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:10.836437 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:10.836439 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:10.836646 (+ 207us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:10.836846 (+ 200us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:10.836847 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:10.836847 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:10.836928 (+ 81us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:10.837167 (+ 239us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:10.837172 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:10.837297 (+ 125us) client_negotiation.cc:770] Sending connection context
0429 14:04:10.837408 (+ 111us) client_negotiation.cc:241] Negotiation successful
0429 14:04:10.837544 (+ 136us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":91,"threads_started":1}
I20260429 14:04:10.838389 7724 heartbeater.cc:344] Connected to a master server at 127.2.197.254:43453
I20260429 14:04:10.838482 7724 heartbeater.cc:461] Registering TS with master...
I20260429 14:04:10.838721 7724 heartbeater.cc:507] Master 127.2.197.254:43453 requested a full tablet report, sending...
I20260429 14:04:10.839247 7531 ts_manager.cc:194] Registered new tserver with Master: 5ea270dcbade4481a05fb51e3ac91671 (127.2.197.193:33607)
I20260429 14:04:10.839926 7531 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.193@KRBTEST.COM'} at 127.2.197.193:58573
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:04:10.885699 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.194:0
--local_ip_for_outbound_sockets=127.2.197.194
--webserver_interface=127.2.197.194
--webserver_port=0
--tserver_master_addrs=127.2.197.254:43453
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:35141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:04:10.963868 7731 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:10.964087 7731 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:10.964126 7731 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:10.965535 7731 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:10.965624 7731 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:10.965667 7731 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.194
I20260429 14:04:10.967288 7731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:35141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.197.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:43453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.194
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:10.967569 7731 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:10.967757 7731 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:10.970144 7739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:10.970230 7736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:10.970270 7731 server_base.cc:1061] running on GCE node
W20260429 14:04:10.970367 7737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:10.970544 7731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:10.970798 7731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:10.971989 7731 hybrid_clock.cc:648] HybridClock initialized: now 1777471450971973 us; error 43 us; skew 500 ppm
Apr 29 14:04:10 dist-test-slave-8n77 krb5kdc[7485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471450, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:10.974407 7731 init.cc:377] Logged in from keytab as kudu/127.2.197.194@KRBTEST.COM (short username kudu)
I20260429 14:04:10.974678 7731 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:10.974753 7731 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:10.974902 7731 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:04:10.975863 7731 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-1/data/instance:
uuid: "658994c3713e40288943f250c32fa6b7"
format_stamp: "Formatted at 2026-04-29 14:04:10 on dist-test-slave-8n77"
server_key: "d9dfb4242087b8ace73ff92ecaba6728"
server_key_iv: "4a9a7cc32d0666b905ea7d569532f39c"
server_key_version: "encryptionkey@0"
I20260429 14:04:10.976186 7731 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance:
uuid: "658994c3713e40288943f250c32fa6b7"
format_stamp: "Formatted at 2026-04-29 14:04:10 on dist-test-slave-8n77"
server_key: "d9dfb4242087b8ace73ff92ecaba6728"
server_key_iv: "4a9a7cc32d0666b905ea7d569532f39c"
server_key_version: "encryptionkey@0"
I20260429 14:04:10.977345 7731 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260429 14:04:10.978084 7745 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:10.978260 7731 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:10.978371 7731 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-1/wal
uuid: "658994c3713e40288943f250c32fa6b7"
format_stamp: "Formatted at 2026-04-29 14:04:10 on dist-test-slave-8n77"
server_key: "d9dfb4242087b8ace73ff92ecaba6728"
server_key_iv: "4a9a7cc32d0666b905ea7d569532f39c"
server_key_version: "encryptionkey@0"
I20260429 14:04:10.978453 7731 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:10.990859 7731 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:10.992558 7731 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:10.992715 7731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:10.992942 7731 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:04:10.993264 7731 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:04:10.993319 7731 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:10.993381 7731 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:04:10.993423 7731 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:10.999629 7731 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.194:42419
I20260429 14:04:10.999697 7858 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.194:42419 every 8 connection(s)
I20260429 14:04:10.999941 7731 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
I20260429 14:04:11.000224 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 7731
I20260429 14:04:11.000351 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance
I20260429 14:04:11.000566 2839 external_mini_cluster.cc:1468] Setting key f3f59e0e0aad9286cd15d304e0904d02
Apr 29 14:04:11 dist-test-slave-8n77 krb5kdc[7485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471450, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.197.195@KRBTEST.COM; defaulting to no policy
I20260429 14:04:11.009931 7572 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:11.000755 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:35513 (local address 127.2.197.254:43453)
0429 14:04:11.000817 (+ 62us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:11.000820 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:11.000998 (+ 178us) server_negotiation.cc:408] Connection header received
0429 14:04:11.001778 (+ 780us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:11.001779 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:11.001790 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:11.001844 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:11.003241 (+ 1397us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:11.003848 (+ 607us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:11.004680 (+ 832us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:11.004805 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:11.007079 (+ 2274us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:11.007084 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:11.007084 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:11.007092 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:11.008610 (+ 1518us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:11.009015 (+ 405us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:11.009016 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:11.009017 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:11.009048 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:11.009278 (+ 230us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:11.009279 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:11.009280 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:11.009379 (+ 99us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:11.009455 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:11.009736 (+ 281us) server_negotiation.cc:300] Negotiation successful
0429 14:04:11.009838 (+ 102us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260429 14:04:11.010051 7861 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:11.000664 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:43453 (local address 127.2.197.194:35513)
0429 14:04:11.000888 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:11.000916 (+ 28us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:11.001671 (+ 755us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:11.001900 (+ 229us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:11.001905 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:11.002362 (+ 457us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:11.003133 (+ 771us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:11.003137 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:11.003946 (+ 809us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:11.003948 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:11.004577 (+ 629us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:11.004580 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:11.004722 (+ 142us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:11.005254 (+ 532us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:11.005267 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:11.006982 (+ 1715us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:11.008701 (+ 1719us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:11.008706 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:11.008709 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:11.008920 (+ 211us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:11.009114 (+ 194us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:11.009115 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:11.009115 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:11.009197 (+ 82us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:11.009444 (+ 247us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:11.009450 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:11.009601 (+ 151us) client_negotiation.cc:770] Sending connection context
0429 14:04:11.009716 (+ 115us) client_negotiation.cc:241] Negotiation successful
0429 14:04:11.009862 (+ 146us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":99,"threads_started":1}
I20260429 14:04:11.010598 7859 heartbeater.cc:344] Connected to a master server at 127.2.197.254:43453
I20260429 14:04:11.010699 7859 heartbeater.cc:461] Registering TS with master...
I20260429 14:04:11.010869 7859 heartbeater.cc:507] Master 127.2.197.254:43453 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.195@KRBTEST.COM" created.
I20260429 14:04:11.011356 7531 ts_manager.cc:194] Registered new tserver with Master: 658994c3713e40288943f250c32fa6b7 (127.2.197.194:42419)
I20260429 14:04:11.011776 7531 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.194@KRBTEST.COM'} at 127.2.197.194:35513
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:04:11.057894 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.195:0
--local_ip_for_outbound_sockets=127.2.197.195
--webserver_interface=127.2.197.195
--webserver_port=0
--tserver_master_addrs=127.2.197.254:43453
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:35141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:04:11.139273 7866 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:11.139448 7866 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:11.139472 7866 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:11.140851 7866 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:11.140892 7866 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:11.140918 7866 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.195
I20260429 14:04:11.142547 7866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:35141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.197.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:43453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.195
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:11.142729 7866 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:11.142859 7866 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:11.145370 7871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:11.145372 7872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:11.145373 7874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:11.145622 7866 server_base.cc:1061] running on GCE node
I20260429 14:04:11.145839 7866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:11.146064 7866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:11.147220 7866 hybrid_clock.cc:648] HybridClock initialized: now 1777471451147207 us; error 37 us; skew 500 ppm
Apr 29 14:04:11 dist-test-slave-8n77 krb5kdc[7485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471451, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:11.149691 7866 init.cc:377] Logged in from keytab as kudu/127.2.197.195@KRBTEST.COM (short username kudu)
I20260429 14:04:11.150007 7866 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:11.150077 7866 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:11.150200 7866 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:04:11.151201 7866 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-2/data/instance:
uuid: "f8f74722fea54e6ea464c173dc35fb61"
format_stamp: "Formatted at 2026-04-29 14:04:11 on dist-test-slave-8n77"
server_key: "f5dd8bc9638cff4cac8a63d91a37b1c8"
server_key_iv: "c48044e91dfe7229b5b33d830e08062e"
server_key_version: "encryptionkey@0"
I20260429 14:04:11.151530 7866 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance:
uuid: "f8f74722fea54e6ea464c173dc35fb61"
format_stamp: "Formatted at 2026-04-29 14:04:11 on dist-test-slave-8n77"
server_key: "f5dd8bc9638cff4cac8a63d91a37b1c8"
server_key_iv: "c48044e91dfe7229b5b33d830e08062e"
server_key_version: "encryptionkey@0"
I20260429 14:04:11.152801 7866 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260429 14:04:11.153759 7880 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:11.153961 7866 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:11.154052 7866 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-2/wal
uuid: "f8f74722fea54e6ea464c173dc35fb61"
format_stamp: "Formatted at 2026-04-29 14:04:11 on dist-test-slave-8n77"
server_key: "f5dd8bc9638cff4cac8a63d91a37b1c8"
server_key_iv: "c48044e91dfe7229b5b33d830e08062e"
server_key_version: "encryptionkey@0"
I20260429 14:04:11.154125 7866 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:11.174418 7866 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:11.176678 7866 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:11.176862 7866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:11.177093 7866 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:04:11.177408 7866 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:04:11.177459 7866 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:11.177500 7866 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:04:11.177524 7866 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:11.183573 7866 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.195:37811
I20260429 14:04:11.183653 7993 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.195:37811 every 8 connection(s)
I20260429 14:04:11.183878 7866 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
Apr 29 14:04:11 dist-test-slave-8n77 krb5kdc[7485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471451, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:04:11.192134 7572 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:11.184665 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:52307 (local address 127.2.197.254:43453)
0429 14:04:11.184749 (+ 84us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:11.184752 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:11.184903 (+ 151us) server_negotiation.cc:408] Connection header received
0429 14:04:11.185411 (+ 508us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:11.185412 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:11.185426 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:11.185472 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:11.186529 (+ 1057us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:11.187020 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:11.187666 (+ 646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:11.187849 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:11.189643 (+ 1794us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:11.189651 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:11.189651 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:11.189657 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:11.190891 (+ 1234us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:11.191289 (+ 398us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:11.191290 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:11.191291 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:11.191319 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:11.191531 (+ 212us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:11.191532 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:11.191533 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:11.191628 (+ 95us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:11.191705 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:11.191953 (+ 248us) server_negotiation.cc:300] Negotiation successful
0429 14:04:11.192042 (+ 89us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260429 14:04:11.192232 7996 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:11.184635 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:43453 (local address 127.2.197.195:52307)
0429 14:04:11.184795 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:11.184824 (+ 29us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:11.185305 (+ 481us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:11.185559 (+ 254us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:11.185563 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:11.185846 (+ 283us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:11.186429 (+ 583us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:11.186432 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:11.187105 (+ 673us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:11.187106 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:11.187562 (+ 456us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:11.187564 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:11.187707 (+ 143us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:11.188190 (+ 483us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:11.188199 (+ 9us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:11.189545 (+ 1346us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:11.190992 (+ 1447us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:11.190997 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:11.191000 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:11.191197 (+ 197us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:11.191382 (+ 185us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:11.191382 (+ 0us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:11.191383 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:11.191463 (+ 80us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:11.191695 (+ 232us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:11.191699 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:11.191823 (+ 124us) client_negotiation.cc:770] Sending connection context
0429 14:04:11.191934 (+ 111us) client_negotiation.cc:241] Negotiation successful
0429 14:04:11.192061 (+ 127us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
I20260429 14:04:11.192842 7994 heartbeater.cc:344] Connected to a master server at 127.2.197.254:43453
I20260429 14:04:11.192934 7994 heartbeater.cc:461] Registering TS with master...
I20260429 14:04:11.193094 7994 heartbeater.cc:507] Master 127.2.197.254:43453 requested a full tablet report, sending...
I20260429 14:04:11.193130 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 7866
I20260429 14:04:11.193198 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance
I20260429 14:04:11.193393 2839 external_mini_cluster.cc:1468] Setting key dff7a1e349a6d56686a049f3301d9be2
I20260429 14:04:11.193580 7531 ts_manager.cc:194] Registered new tserver with Master: f8f74722fea54e6ea464c173dc35fb61 (127.2.197.195:37811)
I20260429 14:04:11.194244 7531 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.195@KRBTEST.COM'} at 127.2.197.195:52307
I20260429 14:04:11.194389 2839 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260429 14:04:11.200893 7572 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:11.195748 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60048 (local address 127.2.197.254:43453)
0429 14:04:11.195816 (+ 68us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:11.195819 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:11.195970 (+ 151us) server_negotiation.cc:408] Connection header received
0429 14:04:11.196010 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:11.196011 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:11.196019 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:11.196069 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:11.196685 (+ 616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:11.197169 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:11.197748 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:11.197863 (+ 115us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:11.198834 (+ 971us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:11.198837 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:11.198838 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:11.198844 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:11.199998 (+ 1154us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:11.200289 (+ 291us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:11.200290 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:11.200291 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:11.200313 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:11.200448 (+ 135us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:11.200448 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:11.200449 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:11.200555 (+ 106us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:11.200625 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:11.200700 (+ 75us) server_negotiation.cc:300] Negotiation successful
0429 14:04:11.200771 (+ 71us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260429 14:04:11.202617 7531 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60048:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 14:04:11.203858 7531 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 14:04:11.209532 8006 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:11.206097 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:37811 (local address 127.0.0.1:47926)
0429 14:04:11.206258 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:11.206315 (+ 57us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:11.206385 (+ 70us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:11.206766 (+ 381us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:11.206768 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:11.206770 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:04:11.207094 (+ 324us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:11.207096 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:11.208277 (+ 1181us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:11.208279 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:11.209237 (+ 958us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:11.209238 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:11.209310 (+ 72us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:11.209343 (+ 33us) client_negotiation.cc:770] Sending connection context
0429 14:04:11.209393 (+ 50us) client_negotiation.cc:241] Negotiation successful
0429 14:04:11.209436 (+ 43us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":76,"threads_started":1}
I20260429 14:04:11.209532 8005 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:11.206097 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:33607 (local address 127.0.0.1:43600)
0429 14:04:11.206469 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:11.206479 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:11.206529 (+ 50us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:11.206760 (+ 231us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:11.206763 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:11.206765 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:04:11.207114 (+ 349us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:11.207117 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:11.208389 (+ 1272us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:11.208390 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:11.209236 (+ 846us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:11.209238 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:11.209308 (+ 70us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:11.209343 (+ 35us) client_negotiation.cc:770] Sending connection context
0429 14:04:11.209393 (+ 50us) client_negotiation.cc:241] Negotiation successful
0429 14:04:11.209436 (+ 43us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":3,"thread_start_us":61,"threads_started":1}
I20260429 14:04:11.210110 8007 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:11.206197 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:43600 (local address 127.2.197.193:33607)
0429 14:04:11.206386 (+ 189us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:11.206393 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:11.206602 (+ 209us) server_negotiation.cc:408] Connection header received
0429 14:04:11.206646 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:11.206648 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:11.206671 (+ 23us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:11.206750 (+ 79us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:04:11.207196 (+ 446us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:11.208310 (+ 1114us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:11.209352 (+ 1042us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:11.209915 (+ 563us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:11.209964 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:11.209993 (+ 29us) server_negotiation.cc:300] Negotiation successful
0429 14:04:11.210033 (+ 40us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":115,"threads_started":1}
I20260429 14:04:11.210387 8010 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:11.206562 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:42419 (local address 127.0.0.1:57068)
0429 14:04:11.206876 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:11.206887 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:11.206947 (+ 60us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:11.207386 (+ 439us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:11.207387 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:11.207389 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:04:11.207587 (+ 198us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:11.207588 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:11.208762 (+ 1174us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:11.208763 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:11.209377 (+ 614us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:11.209379 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:11.209447 (+ 68us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:11.209452 (+ 5us) client_negotiation.cc:770] Sending connection context
0429 14:04:11.209484 (+ 32us) client_negotiation.cc:241] Negotiation successful
0429 14:04:11.209519 (+ 35us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":65,"threads_started":1}
I20260429 14:04:11.210386 8008 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:11.206197 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:47926 (local address 127.2.197.195:37811)
0429 14:04:11.206380 (+ 183us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:11.206392 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:11.206404 (+ 12us) server_negotiation.cc:408] Connection header received
0429 14:04:11.206445 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:11.206446 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:11.206460 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:11.207192 (+ 732us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:04:11.207279 (+ 87us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:11.208190 (+ 911us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:11.209716 (+ 1526us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:11.210116 (+ 400us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:11.210158 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:11.210187 (+ 29us) server_negotiation.cc:300] Negotiation successful
0429 14:04:11.210260 (+ 73us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":105,"threads_started":1}
I20260429 14:04:11.210758 7658 tablet_service.cc:1511] Processing CreateTablet for tablet 7f762be99ccd4cba918b6c6676aca0aa (DEFAULT_TABLE table=test-table [id=0c1b5bf244d24a63b108d226339f263d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:04:11.210822 7928 tablet_service.cc:1511] Processing CreateTablet for tablet 7f762be99ccd4cba918b6c6676aca0aa (DEFAULT_TABLE table=test-table [id=0c1b5bf244d24a63b108d226339f263d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:04:11.211071 7928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f762be99ccd4cba918b6c6676aca0aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:11.211072 7658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f762be99ccd4cba918b6c6676aca0aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:11.211185 8009 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:11.206413 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57068 (local address 127.2.197.194:42419)
0429 14:04:11.206797 (+ 384us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:11.206805 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:11.207215 (+ 410us) server_negotiation.cc:408] Connection header received
0429 14:04:11.207261 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:11.207264 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:11.207282 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:11.207366 (+ 84us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:04:11.207669 (+ 303us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:11.208677 (+ 1008us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:11.210114 (+ 1437us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:11.211008 (+ 894us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:11.211045 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:11.211075 (+ 30us) server_negotiation.cc:300] Negotiation successful
0429 14:04:11.211120 (+ 45us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":93,"threads_started":1}
I20260429 14:04:11.213379 8012 tablet_bootstrap.cc:492] T 7f762be99ccd4cba918b6c6676aca0aa P 5ea270dcbade4481a05fb51e3ac91671: Bootstrap starting.
I20260429 14:04:11.213919 7792 tablet_service.cc:1511] Processing CreateTablet for tablet 7f762be99ccd4cba918b6c6676aca0aa (DEFAULT_TABLE table=test-table [id=0c1b5bf244d24a63b108d226339f263d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:04:11.214136 8012 tablet_bootstrap.cc:654] T 7f762be99ccd4cba918b6c6676aca0aa P 5ea270dcbade4481a05fb51e3ac91671: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:11.214181 7792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f762be99ccd4cba918b6c6676aca0aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:11.214449 8011 tablet_bootstrap.cc:492] T 7f762be99ccd4cba918b6c6676aca0aa P f8f74722fea54e6ea464c173dc35fb61: Bootstrap starting.
I20260429 14:04:11.214500 8012 log.cc:826] T 7f762be99ccd4cba918b6c6676aca0aa P 5ea270dcbade4481a05fb51e3ac91671: Log is configured to *not* fsync() on all Append() calls
I20260429 14:04:11.214995 8011 tablet_bootstrap.cc:654] T 7f762be99ccd4cba918b6c6676aca0aa P f8f74722fea54e6ea464c173dc35fb61: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:11.215272 8012 tablet_bootstrap.cc:492] T 7f762be99ccd4cba918b6c6676aca0aa P 5ea270dcbade4481a05fb51e3ac91671: No bootstrap required, opened a new log
I20260429 14:04:11.215279 8011 log.cc:826] T 7f762be99ccd4cba918b6c6676aca0aa P f8f74722fea54e6ea464c173dc35fb61: Log is configured to *not* fsync() on all Append() calls
I20260429 14:04:11.215337 8012 ts_tablet_manager.cc:1403] T 7f762be99ccd4cba918b6c6676aca0aa P 5ea270dcbade4481a05fb51e3ac91671: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260429 14:04:11.216106 8011 tablet_bootstrap.cc:492] T 7f762be99ccd4cba918b6c6676aca0aa P f8f74722fea54e6ea464c173dc35fb61: No bootstrap required, opened a new log
I20260429 14:04:11.216176 8011 ts_tablet_manager.cc:1403] T 7f762be99ccd4cba918b6c6676aca0aa P f8f74722fea54e6ea464c173dc35fb61: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260429 14:04:11.216794 8015 tablet_bootstrap.cc:492] T 7f762be99ccd4cba918b6c6676aca0aa P 658994c3713e40288943f250c32fa6b7: Bootstrap starting.
I20260429 14:04:11.217334 8015 tablet_bootstrap.cc:654] T 7f762be99ccd4cba918b6c6676aca0aa P 658994c3713e40288943f250c32fa6b7: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:11.217429 8012 raft_consensus.cc:359] T 7f762be99ccd4cba918b6c6676aca0aa P 5ea270dcbade4481a05fb51e3ac91671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea270dcbade4481a05fb51e3ac91671" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 33607 } } peers { permanent_uuid: "f8f74722fea54e6ea464c173dc35fb61" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37811 } } peers { permanent_uuid: "658994c3713e40288943f250c32fa6b7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 42419 } }
I20260429 14:04:11.217576 8012 raft_consensus.cc:385] T 7f762be99ccd4cba918b6c6676aca0aa P 5ea270dcbade4481a05fb51e3ac91671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:11.217609 8012 raft_consensus.cc:740] T 7f762be99ccd4cba918b6c6676aca0aa P 5ea270dcbade4481a05fb51e3ac91671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ea270dcbade4481a05fb51e3ac91671, State: Initialized, Role: FOLLOWER
I20260429 14:04:11.217639 8015 log.cc:826] T 7f762be99ccd4cba918b6c6676aca0aa P 658994c3713e40288943f250c32fa6b7: Log is configured to *not* fsync() on all Append() calls
I20260429 14:04:11.217722 8012 consensus_queue.cc:260] T 7f762be99ccd4cba918b6c6676aca0aa P 5ea270dcbade4481a05fb51e3ac91671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea270dcbade4481a05fb51e3ac91671" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 33607 } } peers { permanent_uuid: "f8f74722fea54e6ea464c173dc35fb61" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37811 } } peers { permanent_uuid: "658994c3713e40288943f250c32fa6b7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 42419 } }
I20260429 14:04:11.218014 8012 ts_tablet_manager.cc:1434] T 7f762be99ccd4cba918b6c6676aca0aa P 5ea270dcbade4481a05fb51e3ac91671: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260429 14:04:11.217957 8011 raft_consensus.cc:359] T 7f762be99ccd4cba918b6c6676aca0aa P f8f74722fea54e6ea464c173dc35fb61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea270dcbade4481a05fb51e3ac91671" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 33607 } } peers { permanent_uuid: "f8f74722fea54e6ea464c173dc35fb61" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37811 } } peers { permanent_uuid: "658994c3713e40288943f250c32fa6b7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 42419 } }
I20260429 14:04:11.218057 7724 heartbeater.cc:499] Master 127.2.197.254:43453 was elected leader, sending a full tablet report...
I20260429 14:04:11.218075 8011 raft_consensus.cc:385] T 7f762be99ccd4cba918b6c6676aca0aa P f8f74722fea54e6ea464c173dc35fb61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:11.218112 8011 raft_consensus.cc:740] T 7f762be99ccd4cba918b6c6676aca0aa P f8f74722fea54e6ea464c173dc35fb61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8f74722fea54e6ea464c173dc35fb61, State: Initialized, Role: FOLLOWER
I20260429 14:04:11.218205 8011 consensus_queue.cc:260] T 7f762be99ccd4cba918b6c6676aca0aa P f8f74722fea54e6ea464c173dc35fb61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea270dcbade4481a05fb51e3ac91671" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 33607 } } peers { permanent_uuid: "f8f74722fea54e6ea464c173dc35fb61" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37811 } } peers { permanent_uuid: "658994c3713e40288943f250c32fa6b7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 42419 } }
I20260429 14:04:11.218478 8011 ts_tablet_manager.cc:1434] T 7f762be99ccd4cba918b6c6676aca0aa P f8f74722fea54e6ea464c173dc35fb61: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260429 14:04:11.218531 7994 heartbeater.cc:499] Master 127.2.197.254:43453 was elected leader, sending a full tablet report...
I20260429 14:04:11.219060 8015 tablet_bootstrap.cc:492] T 7f762be99ccd4cba918b6c6676aca0aa P 658994c3713e40288943f250c32fa6b7: No bootstrap required, opened a new log
I20260429 14:04:11.219147 8015 ts_tablet_manager.cc:1403] T 7f762be99ccd4cba918b6c6676aca0aa P 658994c3713e40288943f250c32fa6b7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260429 14:04:11.220655 8015 raft_consensus.cc:359] T 7f762be99ccd4cba918b6c6676aca0aa P 658994c3713e40288943f250c32fa6b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea270dcbade4481a05fb51e3ac91671" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 33607 } } peers { permanent_uuid: "f8f74722fea54e6ea464c173dc35fb61" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37811 } } peers { permanent_uuid: "658994c3713e40288943f250c32fa6b7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 42419 } }
I20260429 14:04:11.220760 8015 raft_consensus.cc:385] T 7f762be99ccd4cba918b6c6676aca0aa P 658994c3713e40288943f250c32fa6b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:11.220779 8015 raft_consensus.cc:740] T 7f762be99ccd4cba918b6c6676aca0aa P 658994c3713e40288943f250c32fa6b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 658994c3713e40288943f250c32fa6b7, State: Initialized, Role: FOLLOWER
I20260429 14:04:11.220881 8015 consensus_queue.cc:260] T 7f762be99ccd4cba918b6c6676aca0aa P 658994c3713e40288943f250c32fa6b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea270dcbade4481a05fb51e3ac91671" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 33607 } } peers { permanent_uuid: "f8f74722fea54e6ea464c173dc35fb61" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37811 } } peers { permanent_uuid: "658994c3713e40288943f250c32fa6b7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 42419 } }
I20260429 14:04:11.221117 8015 ts_tablet_manager.cc:1434] T 7f762be99ccd4cba918b6c6676aca0aa P 658994c3713e40288943f250c32fa6b7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260429 14:04:11.221208 7859 heartbeater.cc:499] Master 127.2.197.254:43453 was elected leader, sending a full tablet report...
W20260429 14:04:11.250481 7860 tablet.cc:2404] T 7f762be99ccd4cba918b6c6676aca0aa P 658994c3713e40288943f250c32fa6b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260429 14:04:11.329113 7725 tablet.cc:2404] T 7f762be99ccd4cba918b6c6676aca0aa P 5ea270dcbade4481a05fb51e3ac91671: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260429 14:04:11.434543 7995 tablet.cc:2404] T 7f762be99ccd4cba918b6c6676aca0aa P f8f74722fea54e6ea464c173dc35fb61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 14:04:11.476591 8019 raft_consensus.cc:493] T 7f762be99ccd4cba918b6c6676aca0aa P 658994c3713e40288943f250c32fa6b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:04:11.476764 8019 raft_consensus.cc:515] T 7f762be99ccd4cba918b6c6676aca0aa P 658994c3713e40288943f250c32fa6b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea270dcbade4481a05fb51e3ac91671" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 33607 } } peers { permanent_uuid: "f8f74722fea54e6ea464c173dc35fb61" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37811 } } peers { permanent_uuid: "658994c3713e40288943f250c32fa6b7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 42419 } }
I20260429 14:04:11.477152 8019 leader_election.cc:290] T 7f762be99ccd4cba918b6c6676aca0aa P 658994c3713e40288943f250c32fa6b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ea270dcbade4481a05fb51e3ac91671 (127.2.197.193:33607), f8f74722fea54e6ea464c173dc35fb61 (127.2.197.195:37811)
I20260429 14:04:11.480103 7861 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:11.477383 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:33607 (local address 127.2.197.194:59495)
0429 14:04:11.477444 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:11.477457 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:11.477511 (+ 54us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:11.477753 (+ 242us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:11.477755 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:11.477757 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:04:11.477974 (+ 217us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:11.477976 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:11.478925 (+ 949us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:11.478926 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:11.479841 (+ 915us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:11.479844 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:11.479923 (+ 79us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:11.479928 (+ 5us) client_negotiation.cc:770] Sending connection context
0429 14:04:11.479964 (+ 36us) client_negotiation.cc:241] Negotiation successful
0429 14:04:11.480009 (+ 45us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260429 14:04:11.480350 8020 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:11.477415 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:37811 (local address 127.2.197.194:59741)
0429 14:04:11.477614 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:11.477624 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:11.477688 (+ 64us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:11.477894 (+ 206us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:11.477895 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:11.477897 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:04:11.478180 (+ 283us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:11.478182 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:11.479046 (+ 864us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:11.479047 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:11.479851 (+ 804us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:11.479852 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:11.479955 (+ 103us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:11.479958 (+ 3us) client_negotiation.cc:770] Sending connection context
0429 14:04:11.479992 (+ 34us) client_negotiation.cc:241] Negotiation successful
0429 14:04:11.480017 (+ 25us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":122,"threads_started":1}
I20260429 14:04:11.480770 8007 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:11.477478 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:59495 (local address 127.2.197.193:33607)
0429 14:04:11.477552 (+ 74us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:11.477554 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:11.477564 (+ 10us) server_negotiation.cc:408] Connection header received
0429 14:04:11.477596 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:11.477597 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:11.477608 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:11.477683 (+ 75us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:04:11.478060 (+ 377us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:11.478834 (+ 774us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:11.480062 (+ 1228us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:11.480648 (+ 586us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:11.480660 (+ 12us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:11.480686 (+ 26us) server_negotiation.cc:300] Negotiation successful
0429 14:04:11.480724 (+ 38us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260429 14:04:11.481034 8008 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:11.477489 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:59741 (local address 127.2.197.195:37811)
0429 14:04:11.477534 (+ 45us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:11.477536 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:11.477717 (+ 181us) server_negotiation.cc:408] Connection header received
0429 14:04:11.477757 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:11.477759 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:11.477770 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:11.477848 (+ 78us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:04:11.478297 (+ 449us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:11.478972 (+ 675us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:11.479994 (+ 1022us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:11.480882 (+ 888us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:11.480892 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:11.480920 (+ 28us) server_negotiation.cc:300] Negotiation successful
0429 14:04:11.480958 (+ 38us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260429 14:04:11.481263 7678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f762be99ccd4cba918b6c6676aca0aa" candidate_uuid: "658994c3713e40288943f250c32fa6b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ea270dcbade4481a05fb51e3ac91671" is_pre_election: true
I20260429 14:04:11.481294 7948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f762be99ccd4cba918b6c6676aca0aa" candidate_uuid: "658994c3713e40288943f250c32fa6b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8f74722fea54e6ea464c173dc35fb61" is_pre_election: true
I20260429 14:04:11.481549 7678 raft_consensus.cc:2468] T 7f762be99ccd4cba918b6c6676aca0aa P 5ea270dcbade4481a05fb51e3ac91671 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 658994c3713e40288943f250c32fa6b7 in term 0.
I20260429 14:04:11.481613 7948 raft_consensus.cc:2468] T 7f762be99ccd4cba918b6c6676aca0aa P f8f74722fea54e6ea464c173dc35fb61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 658994c3713e40288943f250c32fa6b7 in term 0.
I20260429 14:04:11.481841 7748 leader_election.cc:304] T 7f762be99ccd4cba918b6c6676aca0aa P 658994c3713e40288943f250c32fa6b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ea270dcbade4481a05fb51e3ac91671, 658994c3713e40288943f250c32fa6b7; no voters:
I20260429 14:04:11.481994 8019 raft_consensus.cc:2804] T 7f762be99ccd4cba918b6c6676aca0aa P 658994c3713e40288943f250c32fa6b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 14:04:11.482053 8019 raft_consensus.cc:493] T 7f762be99ccd4cba918b6c6676aca0aa P 658994c3713e40288943f250c32fa6b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 14:04:11.482110 8019 raft_consensus.cc:3060] T 7f762be99ccd4cba918b6c6676aca0aa P 658994c3713e40288943f250c32fa6b7 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:11.482875 8019 raft_consensus.cc:515] T 7f762be99ccd4cba918b6c6676aca0aa P 658994c3713e40288943f250c32fa6b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea270dcbade4481a05fb51e3ac91671" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 33607 } } peers { permanent_uuid: "f8f74722fea54e6ea464c173dc35fb61" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37811 } } peers { permanent_uuid: "658994c3713e40288943f250c32fa6b7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 42419 } }
I20260429 14:04:11.483158 8019 leader_election.cc:290] T 7f762be99ccd4cba918b6c6676aca0aa P 658994c3713e40288943f250c32fa6b7 [CANDIDATE]: Term 1 election: Requested vote from peers 5ea270dcbade4481a05fb51e3ac91671 (127.2.197.193:33607), f8f74722fea54e6ea464c173dc35fb61 (127.2.197.195:37811)
I20260429 14:04:11.483189 7948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f762be99ccd4cba918b6c6676aca0aa" candidate_uuid: "658994c3713e40288943f250c32fa6b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8f74722fea54e6ea464c173dc35fb61"
I20260429 14:04:11.483217 7678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f762be99ccd4cba918b6c6676aca0aa" candidate_uuid: "658994c3713e40288943f250c32fa6b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ea270dcbade4481a05fb51e3ac91671"
I20260429 14:04:11.483289 7948 raft_consensus.cc:3060] T 7f762be99ccd4cba918b6c6676aca0aa P f8f74722fea54e6ea464c173dc35fb61 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:11.483338 7678 raft_consensus.cc:3060] T 7f762be99ccd4cba918b6c6676aca0aa P 5ea270dcbade4481a05fb51e3ac91671 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:11.484053 7948 raft_consensus.cc:2468] T 7f762be99ccd4cba918b6c6676aca0aa P f8f74722fea54e6ea464c173dc35fb61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 658994c3713e40288943f250c32fa6b7 in term 1.
I20260429 14:04:11.484265 7746 leader_election.cc:304] T 7f762be99ccd4cba918b6c6676aca0aa P 658994c3713e40288943f250c32fa6b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 658994c3713e40288943f250c32fa6b7, f8f74722fea54e6ea464c173dc35fb61; no voters:
I20260429 14:04:11.484368 8019 raft_consensus.cc:2804] T 7f762be99ccd4cba918b6c6676aca0aa P 658994c3713e40288943f250c32fa6b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:04:11.484522 8019 raft_consensus.cc:697] T 7f762be99ccd4cba918b6c6676aca0aa P 658994c3713e40288943f250c32fa6b7 [term 1 LEADER]: Becoming Leader. State: Replica: 658994c3713e40288943f250c32fa6b7, State: Running, Role: LEADER
I20260429 14:04:11.484673 8019 consensus_queue.cc:237] T 7f762be99ccd4cba918b6c6676aca0aa P 658994c3713e40288943f250c32fa6b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea270dcbade4481a05fb51e3ac91671" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 33607 } } peers { permanent_uuid: "f8f74722fea54e6ea464c173dc35fb61" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37811 } } peers { permanent_uuid: "658994c3713e40288943f250c32fa6b7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 42419 } }
I20260429 14:04:11.484735 7678 raft_consensus.cc:2468] T 7f762be99ccd4cba918b6c6676aca0aa P 5ea270dcbade4481a05fb51e3ac91671 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 658994c3713e40288943f250c32fa6b7 in term 1.
I20260429 14:04:11.485563 7532 catalog_manager.cc:5671] T 7f762be99ccd4cba918b6c6676aca0aa P 658994c3713e40288943f250c32fa6b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 658994c3713e40288943f250c32fa6b7 (127.2.197.194). New cstate: current_term: 1 leader_uuid: "658994c3713e40288943f250c32fa6b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea270dcbade4481a05fb51e3ac91671" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 33607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8f74722fea54e6ea464c173dc35fb61" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "658994c3713e40288943f250c32fa6b7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 42419 } health_report { overall_health: HEALTHY } } }
I20260429 14:04:11.495173 8009 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:11.492911 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57082 (local address 127.2.197.194:42419)
0429 14:04:11.492978 (+ 67us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:11.492980 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:11.492992 (+ 12us) server_negotiation.cc:408] Connection header received
0429 14:04:11.493099 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:11.493100 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:11.493114 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:11.493182 (+ 68us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:04:11.493452 (+ 270us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:11.493956 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:11.494622 (+ 666us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:11.494786 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:11.494817 (+ 31us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:04:11.494904 (+ 87us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:04:11.494996 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:11.495056 (+ 60us) server_negotiation.cc:300] Negotiation successful
0429 14:04:11.495082 (+ 26us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260429 14:04:11.497236 7678 raft_consensus.cc:1275] T 7f762be99ccd4cba918b6c6676aca0aa P 5ea270dcbade4481a05fb51e3ac91671 [term 1 FOLLOWER]: Refusing update from remote peer 658994c3713e40288943f250c32fa6b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:04:11.497282 7948 raft_consensus.cc:1275] T 7f762be99ccd4cba918b6c6676aca0aa P f8f74722fea54e6ea464c173dc35fb61 [term 1 FOLLOWER]: Refusing update from remote peer 658994c3713e40288943f250c32fa6b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:04:11.497604 8021 consensus_queue.cc:1048] T 7f762be99ccd4cba918b6c6676aca0aa P 658994c3713e40288943f250c32fa6b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ea270dcbade4481a05fb51e3ac91671" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 33607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:04:11.497732 8019 consensus_queue.cc:1048] T 7f762be99ccd4cba918b6c6676aca0aa P 658994c3713e40288943f250c32fa6b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8f74722fea54e6ea464c173dc35fb61" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:04:11.500751 8026 mvcc.cc:204] Tried to move back new op lower bound from 7280523065328996352 to 7280523065281748992. Current Snapshot: MvccSnapshot[applied={T|T < 7280523065328996352 or (T in {7280523065328996352})}]
I20260429 14:04:11.503027 7531 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:04:11.503140 7531 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:04:11.504220 7531 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 4206d9b586cf425e9c6afe0dfd3483c2: Sending DeleteTablet for 3 replicas of tablet 7f762be99ccd4cba918b6c6676aca0aa
I20260429 14:04:11.504653 7658 tablet_service.cc:1558] Processing DeleteTablet for tablet 7f762be99ccd4cba918b6c6676aca0aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:04:11 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:43600
I20260429 14:04:11.504658 7928 tablet_service.cc:1558] Processing DeleteTablet for tablet 7f762be99ccd4cba918b6c6676aca0aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:04:11 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:47926
I20260429 14:04:11.504900 7792 tablet_service.cc:1558] Processing DeleteTablet for tablet 7f762be99ccd4cba918b6c6676aca0aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:04:11 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:57068
I20260429 14:04:11.505139 8036 tablet_replica.cc:333] T 7f762be99ccd4cba918b6c6676aca0aa P 658994c3713e40288943f250c32fa6b7: stopping tablet replica
I20260429 14:04:11.505234 8034 tablet_replica.cc:333] T 7f762be99ccd4cba918b6c6676aca0aa P 5ea270dcbade4481a05fb51e3ac91671: stopping tablet replica
I20260429 14:04:11.505272 8036 raft_consensus.cc:2243] T 7f762be99ccd4cba918b6c6676aca0aa P 658994c3713e40288943f250c32fa6b7 [term 1 LEADER]: Raft consensus shutting down.
I20260429 14:04:11.505376 8034 raft_consensus.cc:2243] T 7f762be99ccd4cba918b6c6676aca0aa P 5ea270dcbade4481a05fb51e3ac91671 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:04:11.505434 8036 raft_consensus.cc:2272] T 7f762be99ccd4cba918b6c6676aca0aa P 658994c3713e40288943f250c32fa6b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:04:11.505520 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 7596
I20260429 14:04:11.505633 8034 raft_consensus.cc:2272] T 7f762be99ccd4cba918b6c6676aca0aa P 5ea270dcbade4481a05fb51e3ac91671 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:04:11.505864 8036 ts_tablet_manager.cc:1916] T 7f762be99ccd4cba918b6c6676aca0aa P 658994c3713e40288943f250c32fa6b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:04:11.506093 8035 tablet_replica.cc:333] T 7f762be99ccd4cba918b6c6676aca0aa P f8f74722fea54e6ea464c173dc35fb61: stopping tablet replica
I20260429 14:04:11.506367 8035 raft_consensus.cc:2243] T 7f762be99ccd4cba918b6c6676aca0aa P f8f74722fea54e6ea464c173dc35fb61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:04:11.506467 8035 raft_consensus.cc:2272] T 7f762be99ccd4cba918b6c6676aca0aa P f8f74722fea54e6ea464c173dc35fb61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:04:11.506898 8035 ts_tablet_manager.cc:1916] T 7f762be99ccd4cba918b6c6676aca0aa P f8f74722fea54e6ea464c173dc35fb61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:04:11.507647 8036 ts_tablet_manager.cc:1929] T 7f762be99ccd4cba918b6c6676aca0aa P 658994c3713e40288943f250c32fa6b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 14:04:11.507714 8036 log.cc:1199] T 7f762be99ccd4cba918b6c6676aca0aa P 658994c3713e40288943f250c32fa6b7: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-1/wal/wals/7f762be99ccd4cba918b6c6676aca0aa
I20260429 14:04:11.507987 8036 ts_tablet_manager.cc:1950] T 7f762be99ccd4cba918b6c6676aca0aa P 658994c3713e40288943f250c32fa6b7: Deleting consensus metadata
I20260429 14:04:11.508392 7518 catalog_manager.cc:5002] TS 658994c3713e40288943f250c32fa6b7 (127.2.197.194:42419): tablet 7f762be99ccd4cba918b6c6676aca0aa (table test-table [id=0c1b5bf244d24a63b108d226339f263d]) successfully deleted
I20260429 14:04:11.508476 8035 ts_tablet_manager.cc:1929] T 7f762be99ccd4cba918b6c6676aca0aa P f8f74722fea54e6ea464c173dc35fb61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 14:04:11.508538 8035 log.cc:1199] T 7f762be99ccd4cba918b6c6676aca0aa P f8f74722fea54e6ea464c173dc35fb61: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777471413505459-2839-0/minicluster-data/ts-2/wal/wals/7f762be99ccd4cba918b6c6676aca0aa
I20260429 14:04:11.508761 8035 ts_tablet_manager.cc:1950] T 7f762be99ccd4cba918b6c6676aca0aa P f8f74722fea54e6ea464c173dc35fb61: Deleting consensus metadata
I20260429 14:04:11.509260 7516 catalog_manager.cc:5002] TS f8f74722fea54e6ea464c173dc35fb61 (127.2.197.195:37811): tablet 7f762be99ccd4cba918b6c6676aca0aa (table test-table [id=0c1b5bf244d24a63b108d226339f263d]) successfully deleted
W20260429 14:04:11.514181 7518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.197.193:33607 (error 108)
I20260429 14:04:11.514346 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 7731
I20260429 14:04:11.515784 8010 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:11.515635 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:33607 (local address 127.0.0.1:43606)
0429 14:04:11.515696 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:11.515741 (+ 45us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.193:33607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260429 14:04:11.515949 7518 catalog_manager.cc:4729] TS 5ea270dcbade4481a05fb51e3ac91671 (127.2.197.193:33607): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7f762be99ccd4cba918b6c6676aca0aa: Network error: Client connection negotiation failed: client connection to 127.2.197.193:33607: connect: Connection refused (error 111)
I20260429 14:04:11.519515 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 7866
I20260429 14:04:11.524247 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 7501
2026-04-29T14:04:11Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3258 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-04-29T14:04:11Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:04:11Z Disabled control of system clock
I20260429 14:04:11.552640 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:35085
--webserver_interface=127.2.197.254
--webserver_port=0
--builtin_ntp_servers=127.2.197.212:45039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:35085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260429 14:04:11.630427 8041 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:11.630614 8041 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:11.630637 8041 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:11.632181 8041 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:04:11.632236 8041 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:11.632248 8041 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:11.632261 8041 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:04:11.632273 8041 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:04:11.633890 8041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:45039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:35085
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:35085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:11.634152 8041 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:11.634424 8041 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:11.637192 8047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:11.637276 8041 server_base.cc:1061] running on GCE node
W20260429 14:04:11.637194 8049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:11.637194 8046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:11.637778 8041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:11.638043 8041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:11.639243 8041 hybrid_clock.cc:648] HybridClock initialized: now 1777471451639234 us; error 48 us; skew 500 ppm
I20260429 14:04:11.640540 8041 webserver.cc:492] Webserver started at http://127.2.197.254:34987/ using document root <none> and password file <none>
I20260429 14:04:11.640786 8041 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:11.640861 8041 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:11.640966 8041 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:04:11.641907 8041 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "506c4acc0faf42c99c3aef713a534de4"
format_stamp: "Formatted at 2026-04-29 14:04:11 on dist-test-slave-8n77"
server_key: "d92438bd6769b9d2958f10091d735ff6"
server_key_iv: "c22285c7f2e9b89364df0246f04536b1"
server_key_version: "encryptionkey@0"
I20260429 14:04:11.642236 8041 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "506c4acc0faf42c99c3aef713a534de4"
format_stamp: "Formatted at 2026-04-29 14:04:11 on dist-test-slave-8n77"
server_key: "d92438bd6769b9d2958f10091d735ff6"
server_key_iv: "c22285c7f2e9b89364df0246f04536b1"
server_key_version: "encryptionkey@0"
I20260429 14:04:11.643504 8041 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:04:11.644392 8055 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:11.644588 8041 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:04:11.644678 8041 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "506c4acc0faf42c99c3aef713a534de4"
format_stamp: "Formatted at 2026-04-29 14:04:11 on dist-test-slave-8n77"
server_key: "d92438bd6769b9d2958f10091d735ff6"
server_key_iv: "c22285c7f2e9b89364df0246f04536b1"
server_key_version: "encryptionkey@0"
I20260429 14:04:11.644760 8041 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:11.648382 8041 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:11.648695 8041 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:11.648823 8041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:11.652454 8041 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:35085
I20260429 14:04:11.652490 8107 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:35085 every 8 connection(s)
I20260429 14:04:11.652793 8041 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:04:11.653447 8108 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:11.655941 8108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 506c4acc0faf42c99c3aef713a534de4: Bootstrap starting.
I20260429 14:04:11.656569 8108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 506c4acc0faf42c99c3aef713a534de4: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:11.656754 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 8041
I20260429 14:04:11.656806 8108 log.cc:826] T 00000000000000000000000000000000 P 506c4acc0faf42c99c3aef713a534de4: Log is configured to *not* fsync() on all Append() calls
I20260429 14:04:11.656888 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:04:11.657097 2839 external_mini_cluster.cc:1468] Setting key f30e12974d4393f8bfa53a23375975dc
I20260429 14:04:11.657567 8108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 506c4acc0faf42c99c3aef713a534de4: No bootstrap required, opened a new log
I20260429 14:04:11.659099 8108 raft_consensus.cc:359] T 00000000000000000000000000000000 P 506c4acc0faf42c99c3aef713a534de4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506c4acc0faf42c99c3aef713a534de4" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 35085 } }
I20260429 14:04:11.659250 8108 raft_consensus.cc:385] T 00000000000000000000000000000000 P 506c4acc0faf42c99c3aef713a534de4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:11.659309 8108 raft_consensus.cc:740] T 00000000000000000000000000000000 P 506c4acc0faf42c99c3aef713a534de4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 506c4acc0faf42c99c3aef713a534de4, State: Initialized, Role: FOLLOWER
I20260429 14:04:11.659443 8108 consensus_queue.cc:260] T 00000000000000000000000000000000 P 506c4acc0faf42c99c3aef713a534de4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506c4acc0faf42c99c3aef713a534de4" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 35085 } }
I20260429 14:04:11.659550 8108 raft_consensus.cc:399] T 00000000000000000000000000000000 P 506c4acc0faf42c99c3aef713a534de4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:04:11.659586 8108 raft_consensus.cc:493] T 00000000000000000000000000000000 P 506c4acc0faf42c99c3aef713a534de4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:04:11.659653 8108 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 506c4acc0faf42c99c3aef713a534de4 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:11.660315 8108 raft_consensus.cc:515] T 00000000000000000000000000000000 P 506c4acc0faf42c99c3aef713a534de4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506c4acc0faf42c99c3aef713a534de4" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 35085 } }
I20260429 14:04:11.660434 8112 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:11.657617 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:50152 (local address 127.2.197.254:35085)
0429 14:04:11.657799 (+ 182us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:11.657806 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:11.657831 (+ 25us) server_negotiation.cc:408] Connection header received
0429 14:04:11.659080 (+ 1249us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:11.659089 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:11.659134 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:11.659292 (+ 158us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:11.660017 (+ 725us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:11.660020 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:11.660020 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:04:11.660025 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:11.660054 (+ 29us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:04:11.660060 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:11.660144 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:11.660217 (+ 73us) server_negotiation.cc:300] Negotiation successful
0429 14:04:11.660259 (+ 42us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
I20260429 14:04:11.660477 8108 leader_election.cc:304] T 00000000000000000000000000000000 P 506c4acc0faf42c99c3aef713a534de4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 506c4acc0faf42c99c3aef713a534de4; no voters:
I20260429 14:04:11.660656 8108 leader_election.cc:290] T 00000000000000000000000000000000 P 506c4acc0faf42c99c3aef713a534de4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:04:11.660687 8113 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 506c4acc0faf42c99c3aef713a534de4 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:04:11.660836 8113 raft_consensus.cc:697] T 00000000000000000000000000000000 P 506c4acc0faf42c99c3aef713a534de4 [term 1 LEADER]: Becoming Leader. State: Replica: 506c4acc0faf42c99c3aef713a534de4, State: Running, Role: LEADER
I20260429 14:04:11.660892 8108 sys_catalog.cc:565] T 00000000000000000000000000000000 P 506c4acc0faf42c99c3aef713a534de4 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:04:11.660986 8113 consensus_queue.cc:237] T 00000000000000000000000000000000 P 506c4acc0faf42c99c3aef713a534de4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506c4acc0faf42c99c3aef713a534de4" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 35085 } }
I20260429 14:04:11.661358 8113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 506c4acc0faf42c99c3aef713a534de4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 506c4acc0faf42c99c3aef713a534de4. Latest consensus state: current_term: 1 leader_uuid: "506c4acc0faf42c99c3aef713a534de4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506c4acc0faf42c99c3aef713a534de4" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 35085 } } }
I20260429 14:04:11.661441 8113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 506c4acc0faf42c99c3aef713a534de4 [sys.catalog]: This master's current role is: LEADER
I20260429 14:04:11.661466 8115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 506c4acc0faf42c99c3aef713a534de4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "506c4acc0faf42c99c3aef713a534de4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506c4acc0faf42c99c3aef713a534de4" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 35085 } } }
I20260429 14:04:11.661571 8115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 506c4acc0faf42c99c3aef713a534de4 [sys.catalog]: This master's current role is: LEADER
W20260429 14:04:11.662807 8129 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 506c4acc0faf42c99c3aef713a534de4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260429 14:04:11.662884 8129 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260429 14:04:11.663077 8120 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:04:11.663360 8120 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:04:11.664825 8120 catalog_manager.cc:1357] Generated new cluster ID: f27f228b3c874c499eb1636e8f08d1a3
I20260429 14:04:11.664884 8120 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:04:11.697057 8120 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:04:11.697253 8120 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:04:11.702073 8120 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 506c4acc0faf42c99c3aef713a534de4: Generated new TSK 0
I20260429 14:04:11.702306 8120 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 14:04:11.705621 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.193:0
--local_ip_for_outbound_sockets=127.2.197.193
--webserver_interface=127.2.197.193
--webserver_port=0
--tserver_master_addrs=127.2.197.254:35085
--builtin_ntp_servers=127.2.197.212:45039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260429 14:04:11.786193 8132 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:11.786441 8132 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:11.786480 8132 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:11.787992 8132 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:11.788064 8132 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:11.788108 8132 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.193
I20260429 14:04:11.789674 8132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:45039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.197.193
--webserver_port=0
--tserver_master_addrs=127.2.197.254:35085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.193
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:11.789933 8132 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:11.790160 8132 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:11.793023 8138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:11.793035 8140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:11.793202 8132 server_base.cc:1061] running on GCE node
W20260429 14:04:11.793325 8137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:11.793566 8132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:11.793819 8132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:11.794991 8132 hybrid_clock.cc:648] HybridClock initialized: now 1777471451794960 us; error 45 us; skew 500 ppm
I20260429 14:04:11.796165 8132 webserver.cc:492] Webserver started at http://127.2.197.193:35369/ using document root <none> and password file <none>
I20260429 14:04:11.796380 8132 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:11.796430 8132 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:11.796545 8132 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:04:11.797479 8132 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-0/data/instance:
uuid: "ffd4fe4c799f4108a282f78c586c8773"
format_stamp: "Formatted at 2026-04-29 14:04:11 on dist-test-slave-8n77"
server_key: "e3b0b55e264a75036654bd919105bc68"
server_key_iv: "d8545f0605f46457ee78d721cd6def5b"
server_key_version: "encryptionkey@0"
I20260429 14:04:11.797823 8132 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance:
uuid: "ffd4fe4c799f4108a282f78c586c8773"
format_stamp: "Formatted at 2026-04-29 14:04:11 on dist-test-slave-8n77"
server_key: "e3b0b55e264a75036654bd919105bc68"
server_key_iv: "d8545f0605f46457ee78d721cd6def5b"
server_key_version: "encryptionkey@0"
I20260429 14:04:11.799120 8132 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260429 14:04:11.800060 8146 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:11.800349 8132 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:04:11.800431 8132 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-0/wal
uuid: "ffd4fe4c799f4108a282f78c586c8773"
format_stamp: "Formatted at 2026-04-29 14:04:11 on dist-test-slave-8n77"
server_key: "e3b0b55e264a75036654bd919105bc68"
server_key_iv: "d8545f0605f46457ee78d721cd6def5b"
server_key_version: "encryptionkey@0"
I20260429 14:04:11.800514 8132 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:11.804102 8132 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:11.804647 8132 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:11.804819 8132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:11.805018 8132 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:04:11.805320 8132 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:04:11.805374 8132 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:11.805418 8132 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:04:11.805440 8132 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:11.814008 8132 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.193:40727
I20260429 14:04:11.814085 8259 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.193:40727 every 8 connection(s)
I20260429 14:04:11.814500 8132 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
I20260429 14:04:11.817400 8112 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:11.815263 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:53411 (local address 127.2.197.254:35085)
0429 14:04:11.815336 (+ 73us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:11.815339 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:11.815631 (+ 292us) server_negotiation.cc:408] Connection header received
0429 14:04:11.816227 (+ 596us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:11.816228 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:11.816234 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:11.816299 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:11.817114 (+ 815us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:11.817115 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:11.817115 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:04:11.817120 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:11.817135 (+ 15us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:04:11.817139 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:11.817193 (+ 54us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:11.817314 (+ 121us) server_negotiation.cc:300] Negotiation successful
0429 14:04:11.817338 (+ 24us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260429 14:04:11.817508 8262 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:11.815234 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:35085 (local address 127.2.197.193:53411)
0429 14:04:11.815437 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:11.815470 (+ 33us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:11.816097 (+ 627us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:11.816382 (+ 285us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:11.816387 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:11.816398 (+ 11us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:11.816983 (+ 585us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 14:04:11.816984 (+ 1us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:11.816998 (+ 14us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 14:04:11.817022 (+ 24us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:11.817209 (+ 187us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:11.817213 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:11.817215 (+ 2us) client_negotiation.cc:770] Sending connection context
0429 14:04:11.817296 (+ 81us) client_negotiation.cc:241] Negotiation successful
0429 14:04:11.817338 (+ 42us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":76,"threads_started":1}
I20260429 14:04:11.817993 8260 heartbeater.cc:344] Connected to a master server at 127.2.197.254:35085
I20260429 14:04:11.818125 8260 heartbeater.cc:461] Registering TS with master...
I20260429 14:04:11.818321 8260 heartbeater.cc:507] Master 127.2.197.254:35085 requested a full tablet report, sending...
I20260429 14:04:11.818987 8072 ts_manager.cc:194] Registered new tserver with Master: ffd4fe4c799f4108a282f78c586c8773 (127.2.197.193:40727)
I20260429 14:04:11.820444 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 8132
I20260429 14:04:11.820528 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance
I20260429 14:04:11.820752 2839 external_mini_cluster.cc:1468] Setting key c99a9f740c605f294c7e97bbbb2f9642
I20260429 14:04:11.821657 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.194:0
--local_ip_for_outbound_sockets=127.2.197.194
--webserver_interface=127.2.197.194
--webserver_port=0
--tserver_master_addrs=127.2.197.254:35085
--builtin_ntp_servers=127.2.197.212:45039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260429 14:04:11.903098 8263 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:11.903295 8263 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:11.903318 8263 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:11.905071 8263 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:11.905120 8263 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:11.905148 8263 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.194
I20260429 14:04:11.906895 8263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:45039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.197.194
--webserver_port=0
--tserver_master_addrs=127.2.197.254:35085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.194
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:11.907145 8263 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:11.907408 8263 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:11.910106 8268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:11.910208 8269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:11.910176 8271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:11.910320 8263 server_base.cc:1061] running on GCE node
I20260429 14:04:11.910606 8263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:11.910832 8263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:11.911962 8263 hybrid_clock.cc:648] HybridClock initialized: now 1777471451911945 us; error 33 us; skew 500 ppm
I20260429 14:04:11.913007 8263 webserver.cc:492] Webserver started at http://127.2.197.194:33119/ using document root <none> and password file <none>
I20260429 14:04:11.913213 8263 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:11.913259 8263 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:11.913367 8263 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:04:11.914412 8263 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-1/data/instance:
uuid: "27bb31df5fa54c49b22e75a37da4aae0"
format_stamp: "Formatted at 2026-04-29 14:04:11 on dist-test-slave-8n77"
server_key: "0ea6f75ea13b5c582761bfcd26b70d69"
server_key_iv: "5683e1aab1da0dc795fdb83076af57a1"
server_key_version: "encryptionkey@0"
I20260429 14:04:11.914743 8263 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance:
uuid: "27bb31df5fa54c49b22e75a37da4aae0"
format_stamp: "Formatted at 2026-04-29 14:04:11 on dist-test-slave-8n77"
server_key: "0ea6f75ea13b5c582761bfcd26b70d69"
server_key_iv: "5683e1aab1da0dc795fdb83076af57a1"
server_key_version: "encryptionkey@0"
I20260429 14:04:11.916062 8263 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:11.916965 8277 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:11.917358 8263 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:11.917433 8263 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-1/wal
uuid: "27bb31df5fa54c49b22e75a37da4aae0"
format_stamp: "Formatted at 2026-04-29 14:04:11 on dist-test-slave-8n77"
server_key: "0ea6f75ea13b5c582761bfcd26b70d69"
server_key_iv: "5683e1aab1da0dc795fdb83076af57a1"
server_key_version: "encryptionkey@0"
I20260429 14:04:11.917479 8263 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:11.920943 8263 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:11.921195 8263 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:11.921314 8263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:11.921507 8263 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:04:11.921813 8263 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:04:11.921865 8263 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:11.921906 8263 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:04:11.921939 8263 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:11.927747 8263 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.194:36605
I20260429 14:04:11.927788 8390 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.194:36605 every 8 connection(s)
I20260429 14:04:11.928089 8263 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
I20260429 14:04:11.930562 8112 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:11.928896 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:52251 (local address 127.2.197.254:35085)
0429 14:04:11.928962 (+ 66us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:11.928964 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:11.929137 (+ 173us) server_negotiation.cc:408] Connection header received
0429 14:04:11.929566 (+ 429us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:11.929568 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:11.929573 (+ 5us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:11.929638 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:11.930270 (+ 632us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:11.930271 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:11.930271 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:04:11.930329 (+ 58us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:11.930352 (+ 23us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:04:11.930356 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:11.930398 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:11.930486 (+ 88us) server_negotiation.cc:300] Negotiation successful
0429 14:04:11.930500 (+ 14us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260429 14:04:11.930661 8393 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:11.928882 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:35085 (local address 127.2.197.194:52251)
0429 14:04:11.929031 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:11.929061 (+ 30us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:11.929486 (+ 425us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:11.929676 (+ 190us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:11.929679 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:11.929687 (+ 8us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:11.930163 (+ 476us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 14:04:11.930164 (+ 1us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:11.930173 (+ 9us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 14:04:11.930191 (+ 18us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:11.930409 (+ 218us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:11.930411 (+ 2us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:11.930413 (+ 2us) client_negotiation.cc:770] Sending connection context
0429 14:04:11.930467 (+ 54us) client_negotiation.cc:241] Negotiation successful
0429 14:04:11.930498 (+ 31us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":71,"threads_started":1}
I20260429 14:04:11.931149 8391 heartbeater.cc:344] Connected to a master server at 127.2.197.254:35085
I20260429 14:04:11.931222 8391 heartbeater.cc:461] Registering TS with master...
I20260429 14:04:11.931437 8391 heartbeater.cc:507] Master 127.2.197.254:35085 requested a full tablet report, sending...
I20260429 14:04:11.931947 8072 ts_manager.cc:194] Registered new tserver with Master: 27bb31df5fa54c49b22e75a37da4aae0 (127.2.197.194:36605)
I20260429 14:04:11.936088 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 8263
I20260429 14:04:11.936172 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance
I20260429 14:04:11.936300 2839 external_mini_cluster.cc:1468] Setting key 248cdd748b1176720d4b95e70c9d2743
I20260429 14:04:11.937232 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.195:0
--local_ip_for_outbound_sockets=127.2.197.195
--webserver_interface=127.2.197.195
--webserver_port=0
--tserver_master_addrs=127.2.197.254:35085
--builtin_ntp_servers=127.2.197.212:45039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260429 14:04:12.013403 8394 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:12.013590 8394 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:12.013613 8394 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:12.015205 8394 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:12.015255 8394 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:12.015281 8394 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.195
I20260429 14:04:12.016839 8394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:45039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.197.195
--webserver_port=0
--tserver_master_addrs=127.2.197.254:35085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.195
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:12.017074 8394 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:12.017316 8394 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:12.020016 8399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:12.020010 8402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:12.020010 8400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:12.020344 8394 server_base.cc:1061] running on GCE node
I20260429 14:04:12.021217 8394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:12.021494 8394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:12.022713 8394 hybrid_clock.cc:648] HybridClock initialized: now 1777471452022664 us; error 40 us; skew 500 ppm
I20260429 14:04:12.024281 8394 webserver.cc:492] Webserver started at http://127.2.197.195:37415/ using document root <none> and password file <none>
I20260429 14:04:12.024552 8394 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:12.024627 8394 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:12.024747 8394 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:04:12.025684 8394 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-2/data/instance:
uuid: "3daab3f5ab2f49aa96c8f3a4926c6ccc"
format_stamp: "Formatted at 2026-04-29 14:04:12 on dist-test-slave-8n77"
server_key: "2834f0d12ea20d2108e151a80a66daeb"
server_key_iv: "ce58917d1b5b51a12f0605edb0bc2460"
server_key_version: "encryptionkey@0"
I20260429 14:04:12.026017 8394 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance:
uuid: "3daab3f5ab2f49aa96c8f3a4926c6ccc"
format_stamp: "Formatted at 2026-04-29 14:04:12 on dist-test-slave-8n77"
server_key: "2834f0d12ea20d2108e151a80a66daeb"
server_key_iv: "ce58917d1b5b51a12f0605edb0bc2460"
server_key_version: "encryptionkey@0"
I20260429 14:04:12.027482 8394 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260429 14:04:12.028524 8408 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:12.028818 8394 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:12.028901 8394 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-2/wal
uuid: "3daab3f5ab2f49aa96c8f3a4926c6ccc"
format_stamp: "Formatted at 2026-04-29 14:04:12 on dist-test-slave-8n77"
server_key: "2834f0d12ea20d2108e151a80a66daeb"
server_key_iv: "ce58917d1b5b51a12f0605edb0bc2460"
server_key_version: "encryptionkey@0"
I20260429 14:04:12.028985 8394 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:12.032410 8394 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:12.032661 8394 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:12.032795 8394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:12.032994 8394 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:04:12.033272 8394 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:04:12.033325 8394 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:12.033365 8394 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:04:12.033390 8394 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:12.039050 8394 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.195:35849
I20260429 14:04:12.039134 8521 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.195:35849 every 8 connection(s)
I20260429 14:04:12.039398 8394 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
I20260429 14:04:12.041497 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 8394
I20260429 14:04:12.041630 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance
I20260429 14:04:12.041808 2839 external_mini_cluster.cc:1468] Setting key 021edafb0488270b22cb7b82204cf0c1
I20260429 14:04:12.042053 8112 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:12.040149 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:51577 (local address 127.2.197.254:35085)
0429 14:04:12.040195 (+ 46us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:12.040197 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:12.040383 (+ 186us) server_negotiation.cc:408] Connection header received
0429 14:04:12.040796 (+ 413us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:12.040797 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:12.040805 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:12.040869 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:12.041696 (+ 827us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:12.041697 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:12.041698 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:04:12.041703 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:12.041720 (+ 17us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:04:12.041727 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:12.041858 (+ 131us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:12.041965 (+ 107us) server_negotiation.cc:300] Negotiation successful
0429 14:04:12.041984 (+ 19us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260429 14:04:12.042177 8524 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:12.040122 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:35085 (local address 127.2.197.195:51577)
0429 14:04:12.040281 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:12.040310 (+ 29us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:12.040720 (+ 410us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:12.040927 (+ 207us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:12.040930 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:12.040939 (+ 9us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:12.041424 (+ 485us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 14:04:12.041424 (+ 0us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:12.041439 (+ 15us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 14:04:12.041466 (+ 27us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:12.041859 (+ 393us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:12.041872 (+ 13us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:12.041875 (+ 3us) client_negotiation.cc:770] Sending connection context
0429 14:04:12.041973 (+ 98us) client_negotiation.cc:241] Negotiation successful
0429 14:04:12.042013 (+ 40us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":83,"threads_started":1}
I20260429 14:04:12.042779 8522 heartbeater.cc:344] Connected to a master server at 127.2.197.254:35085
I20260429 14:04:12.042891 8522 heartbeater.cc:461] Registering TS with master...
I20260429 14:04:12.043021 8522 heartbeater.cc:507] Master 127.2.197.254:35085 requested a full tablet report, sending...
I20260429 14:04:12.043382 8072 ts_manager.cc:194] Registered new tserver with Master: 3daab3f5ab2f49aa96c8f3a4926c6ccc (127.2.197.195:35849)
I20260429 14:04:12.054097 2839 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260429 14:04:12.057098 8112 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:12.055787 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:50160 (local address 127.2.197.254:35085)
0429 14:04:12.055866 (+ 79us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:12.055868 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:12.055957 (+ 89us) server_negotiation.cc:408] Connection header received
0429 14:04:12.056069 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:12.056070 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:12.056075 (+ 5us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:12.056140 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:12.056828 (+ 688us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:12.056829 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:12.056830 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:04:12.056835 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:12.056856 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:04:12.056860 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:12.056922 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:12.056997 (+ 75us) server_negotiation.cc:300] Negotiation successful
0429 14:04:12.057014 (+ 17us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260429 14:04:12.058856 8072 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 14:04:12.060158 8072 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 14:04:12.063901 8531 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:12.062259 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:40727 (local address 127.0.0.1:60686)
0429 14:04:12.062473 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:12.062507 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:12.062573 (+ 66us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:12.062845 (+ 272us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:12.062850 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:12.062857 (+ 7us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:12.063337 (+ 480us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 14:04:12.063337 (+ 0us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:12.063350 (+ 13us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 14:04:12.063371 (+ 21us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:12.063707 (+ 336us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:12.063711 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:12.063715 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 14:04:12.063781 (+ 66us) client_negotiation.cc:241] Negotiation successful
0429 14:04:12.063795 (+ 14us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":98,"threads_started":1}
I20260429 14:04:12.064106 8534 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:12.062274 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:35849 (local address 127.0.0.1:55442)
0429 14:04:12.062651 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:12.062661 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:12.062704 (+ 43us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:12.063020 (+ 316us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:12.063021 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:12.063025 (+ 4us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:12.063501 (+ 476us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 14:04:12.063502 (+ 1us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:12.063509 (+ 7us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 14:04:12.063521 (+ 12us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:12.063763 (+ 242us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:12.063764 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:12.063765 (+ 1us) client_negotiation.cc:770] Sending connection context
0429 14:04:12.063822 (+ 57us) client_negotiation.cc:241] Negotiation successful
0429 14:04:12.063836 (+ 14us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":121,"threads_started":1}
I20260429 14:04:12.064190 8533 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:12.062381 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55442 (local address 127.2.197.195:35849)
0429 14:04:12.062750 (+ 369us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:12.062758 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:12.062767 (+ 9us) server_negotiation.cc:408] Connection header received
0429 14:04:12.062793 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:12.062795 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:12.062801 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:12.062877 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:12.063611 (+ 734us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:12.063616 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:12.063617 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:04:12.063622 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:12.063648 (+ 26us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:04:12.063653 (+ 5us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:12.063732 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:12.063855 (+ 123us) server_negotiation.cc:300] Negotiation successful
0429 14:04:12.063872 (+ 17us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":89,"threads_started":1}
I20260429 14:04:12.064445 8532 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:12.062359 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60686 (local address 127.2.197.193:40727)
0429 14:04:12.062515 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:12.062523 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:12.062568 (+ 45us) server_negotiation.cc:408] Connection header received
0429 14:04:12.062676 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:12.062680 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:12.062699 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:12.063079 (+ 380us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:12.063592 (+ 513us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:12.063595 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:12.063596 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:04:12.063600 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:12.063622 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:04:12.063629 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:12.063713 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:12.063887 (+ 174us) server_negotiation.cc:300] Negotiation successful
0429 14:04:12.063902 (+ 15us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
I20260429 14:04:12.064702 8530 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:12.062496 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:36605 (local address 127.0.0.1:40684)
0429 14:04:12.062899 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:12.062910 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:12.062963 (+ 53us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:12.063602 (+ 639us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:12.063604 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:12.063606 (+ 2us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:12.064327 (+ 721us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 14:04:12.064328 (+ 1us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:12.064334 (+ 6us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 14:04:12.064346 (+ 12us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:12.064582 (+ 236us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:12.064583 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:12.064584 (+ 1us) client_negotiation.cc:770] Sending connection context
0429 14:04:12.064627 (+ 43us) client_negotiation.cc:241] Negotiation successful
0429 14:04:12.064641 (+ 14us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":91,"threads_started":1}
I20260429 14:04:12.064743 8456 tablet_service.cc:1511] Processing CreateTablet for tablet 92926f3d8bf345dc8d3cb784f171d63b (DEFAULT_TABLE table=test-table [id=ec7156f5c0a2423ca4075f7dfa952e4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:04:12.065053 8456 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92926f3d8bf345dc8d3cb784f171d63b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:12.065240 8535 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:12.062728 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:40684 (local address 127.2.197.194:36605)
0429 14:04:12.062913 (+ 185us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:12.062920 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:12.062958 (+ 38us) server_negotiation.cc:408] Connection header received
0429 14:04:12.063264 (+ 306us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:12.063270 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:12.063279 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:12.063339 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:12.064431 (+ 1092us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:12.064437 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:12.064437 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:04:12.064441 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:12.064466 (+ 25us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:04:12.064472 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:12.064554 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:12.064847 (+ 293us) server_negotiation.cc:300] Negotiation successful
0429 14:04:12.064863 (+ 16us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":97,"threads_started":1}
I20260429 14:04:12.065200 8194 tablet_service.cc:1511] Processing CreateTablet for tablet 92926f3d8bf345dc8d3cb784f171d63b (DEFAULT_TABLE table=test-table [id=ec7156f5c0a2423ca4075f7dfa952e4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:04:12.065469 8194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92926f3d8bf345dc8d3cb784f171d63b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:12.065793 8325 tablet_service.cc:1511] Processing CreateTablet for tablet 92926f3d8bf345dc8d3cb784f171d63b (DEFAULT_TABLE table=test-table [id=ec7156f5c0a2423ca4075f7dfa952e4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:04:12.066040 8325 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92926f3d8bf345dc8d3cb784f171d63b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:12.068250 8536 tablet_bootstrap.cc:492] T 92926f3d8bf345dc8d3cb784f171d63b P 3daab3f5ab2f49aa96c8f3a4926c6ccc: Bootstrap starting.
I20260429 14:04:12.068444 8537 tablet_bootstrap.cc:492] T 92926f3d8bf345dc8d3cb784f171d63b P ffd4fe4c799f4108a282f78c586c8773: Bootstrap starting.
I20260429 14:04:12.069199 8537 tablet_bootstrap.cc:654] T 92926f3d8bf345dc8d3cb784f171d63b P ffd4fe4c799f4108a282f78c586c8773: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:12.069211 8536 tablet_bootstrap.cc:654] T 92926f3d8bf345dc8d3cb784f171d63b P 3daab3f5ab2f49aa96c8f3a4926c6ccc: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:12.069226 8538 tablet_bootstrap.cc:492] T 92926f3d8bf345dc8d3cb784f171d63b P 27bb31df5fa54c49b22e75a37da4aae0: Bootstrap starting.
I20260429 14:04:12.069439 8537 log.cc:826] T 92926f3d8bf345dc8d3cb784f171d63b P ffd4fe4c799f4108a282f78c586c8773: Log is configured to *not* fsync() on all Append() calls
I20260429 14:04:12.069489 8536 log.cc:826] T 92926f3d8bf345dc8d3cb784f171d63b P 3daab3f5ab2f49aa96c8f3a4926c6ccc: Log is configured to *not* fsync() on all Append() calls
I20260429 14:04:12.070343 8537 tablet_bootstrap.cc:492] T 92926f3d8bf345dc8d3cb784f171d63b P ffd4fe4c799f4108a282f78c586c8773: No bootstrap required, opened a new log
I20260429 14:04:12.070411 8537 ts_tablet_manager.cc:1403] T 92926f3d8bf345dc8d3cb784f171d63b P ffd4fe4c799f4108a282f78c586c8773: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260429 14:04:12.070570 8538 tablet_bootstrap.cc:654] T 92926f3d8bf345dc8d3cb784f171d63b P 27bb31df5fa54c49b22e75a37da4aae0: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:12.070655 8536 tablet_bootstrap.cc:492] T 92926f3d8bf345dc8d3cb784f171d63b P 3daab3f5ab2f49aa96c8f3a4926c6ccc: No bootstrap required, opened a new log
I20260429 14:04:12.070721 8536 ts_tablet_manager.cc:1403] T 92926f3d8bf345dc8d3cb784f171d63b P 3daab3f5ab2f49aa96c8f3a4926c6ccc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260429 14:04:12.070972 8538 log.cc:826] T 92926f3d8bf345dc8d3cb784f171d63b P 27bb31df5fa54c49b22e75a37da4aae0: Log is configured to *not* fsync() on all Append() calls
I20260429 14:04:12.071888 8537 raft_consensus.cc:359] T 92926f3d8bf345dc8d3cb784f171d63b P ffd4fe4c799f4108a282f78c586c8773 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd4fe4c799f4108a282f78c586c8773" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 40727 } } peers { permanent_uuid: "3daab3f5ab2f49aa96c8f3a4926c6ccc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35849 } } peers { permanent_uuid: "27bb31df5fa54c49b22e75a37da4aae0" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 36605 } }
I20260429 14:04:12.072000 8537 raft_consensus.cc:385] T 92926f3d8bf345dc8d3cb784f171d63b P ffd4fe4c799f4108a282f78c586c8773 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:12.072019 8537 raft_consensus.cc:740] T 92926f3d8bf345dc8d3cb784f171d63b P ffd4fe4c799f4108a282f78c586c8773 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffd4fe4c799f4108a282f78c586c8773, State: Initialized, Role: FOLLOWER
I20260429 14:04:12.072149 8537 consensus_queue.cc:260] T 92926f3d8bf345dc8d3cb784f171d63b P ffd4fe4c799f4108a282f78c586c8773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd4fe4c799f4108a282f78c586c8773" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 40727 } } peers { permanent_uuid: "3daab3f5ab2f49aa96c8f3a4926c6ccc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35849 } } peers { permanent_uuid: "27bb31df5fa54c49b22e75a37da4aae0" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 36605 } }
I20260429 14:04:12.072283 8538 tablet_bootstrap.cc:492] T 92926f3d8bf345dc8d3cb784f171d63b P 27bb31df5fa54c49b22e75a37da4aae0: No bootstrap required, opened a new log
I20260429 14:04:12.072373 8538 ts_tablet_manager.cc:1403] T 92926f3d8bf345dc8d3cb784f171d63b P 27bb31df5fa54c49b22e75a37da4aae0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260429 14:04:12.072388 8537 ts_tablet_manager.cc:1434] T 92926f3d8bf345dc8d3cb784f171d63b P ffd4fe4c799f4108a282f78c586c8773: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260429 14:04:12.072384 8536 raft_consensus.cc:359] T 92926f3d8bf345dc8d3cb784f171d63b P 3daab3f5ab2f49aa96c8f3a4926c6ccc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd4fe4c799f4108a282f78c586c8773" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 40727 } } peers { permanent_uuid: "3daab3f5ab2f49aa96c8f3a4926c6ccc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35849 } } peers { permanent_uuid: "27bb31df5fa54c49b22e75a37da4aae0" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 36605 } }
I20260429 14:04:12.072520 8260 heartbeater.cc:499] Master 127.2.197.254:35085 was elected leader, sending a full tablet report...
I20260429 14:04:12.072528 8536 raft_consensus.cc:385] T 92926f3d8bf345dc8d3cb784f171d63b P 3daab3f5ab2f49aa96c8f3a4926c6ccc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:12.072566 8536 raft_consensus.cc:740] T 92926f3d8bf345dc8d3cb784f171d63b P 3daab3f5ab2f49aa96c8f3a4926c6ccc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3daab3f5ab2f49aa96c8f3a4926c6ccc, State: Initialized, Role: FOLLOWER
I20260429 14:04:12.072691 8536 consensus_queue.cc:260] T 92926f3d8bf345dc8d3cb784f171d63b P 3daab3f5ab2f49aa96c8f3a4926c6ccc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd4fe4c799f4108a282f78c586c8773" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 40727 } } peers { permanent_uuid: "3daab3f5ab2f49aa96c8f3a4926c6ccc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35849 } } peers { permanent_uuid: "27bb31df5fa54c49b22e75a37da4aae0" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 36605 } }
I20260429 14:04:12.072933 8536 ts_tablet_manager.cc:1434] T 92926f3d8bf345dc8d3cb784f171d63b P 3daab3f5ab2f49aa96c8f3a4926c6ccc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260429 14:04:12.072991 8522 heartbeater.cc:499] Master 127.2.197.254:35085 was elected leader, sending a full tablet report...
I20260429 14:04:12.074136 8538 raft_consensus.cc:359] T 92926f3d8bf345dc8d3cb784f171d63b P 27bb31df5fa54c49b22e75a37da4aae0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd4fe4c799f4108a282f78c586c8773" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 40727 } } peers { permanent_uuid: "3daab3f5ab2f49aa96c8f3a4926c6ccc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35849 } } peers { permanent_uuid: "27bb31df5fa54c49b22e75a37da4aae0" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 36605 } }
I20260429 14:04:12.074241 8538 raft_consensus.cc:385] T 92926f3d8bf345dc8d3cb784f171d63b P 27bb31df5fa54c49b22e75a37da4aae0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:12.074259 8538 raft_consensus.cc:740] T 92926f3d8bf345dc8d3cb784f171d63b P 27bb31df5fa54c49b22e75a37da4aae0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27bb31df5fa54c49b22e75a37da4aae0, State: Initialized, Role: FOLLOWER
I20260429 14:04:12.074398 8538 consensus_queue.cc:260] T 92926f3d8bf345dc8d3cb784f171d63b P 27bb31df5fa54c49b22e75a37da4aae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd4fe4c799f4108a282f78c586c8773" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 40727 } } peers { permanent_uuid: "3daab3f5ab2f49aa96c8f3a4926c6ccc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35849 } } peers { permanent_uuid: "27bb31df5fa54c49b22e75a37da4aae0" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 36605 } }
I20260429 14:04:12.074620 8538 ts_tablet_manager.cc:1434] T 92926f3d8bf345dc8d3cb784f171d63b P 27bb31df5fa54c49b22e75a37da4aae0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260429 14:04:12.074677 8391 heartbeater.cc:499] Master 127.2.197.254:35085 was elected leader, sending a full tablet report...
I20260429 14:04:12.145087 8543 raft_consensus.cc:493] T 92926f3d8bf345dc8d3cb784f171d63b P 3daab3f5ab2f49aa96c8f3a4926c6ccc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:04:12.145344 8543 raft_consensus.cc:515] T 92926f3d8bf345dc8d3cb784f171d63b P 3daab3f5ab2f49aa96c8f3a4926c6ccc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd4fe4c799f4108a282f78c586c8773" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 40727 } } peers { permanent_uuid: "3daab3f5ab2f49aa96c8f3a4926c6ccc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35849 } } peers { permanent_uuid: "27bb31df5fa54c49b22e75a37da4aae0" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 36605 } }
I20260429 14:04:12.145741 8543 leader_election.cc:290] T 92926f3d8bf345dc8d3cb784f171d63b P 3daab3f5ab2f49aa96c8f3a4926c6ccc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffd4fe4c799f4108a282f78c586c8773 (127.2.197.193:40727), 27bb31df5fa54c49b22e75a37da4aae0 (127.2.197.194:36605)
I20260429 14:04:12.147356 8524 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:12.146006 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:40727 (local address 127.2.197.195:44521)
0429 14:04:12.146072 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:12.146085 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:12.146157 (+ 72us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:12.146344 (+ 187us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:12.146345 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:12.146348 (+ 3us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:12.146930 (+ 582us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 14:04:12.146931 (+ 1us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:12.146939 (+ 8us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 14:04:12.146953 (+ 14us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:12.147175 (+ 222us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:12.147176 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:12.147177 (+ 1us) client_negotiation.cc:770] Sending connection context
0429 14:04:12.147241 (+ 64us) client_negotiation.cc:241] Negotiation successful
0429 14:04:12.147254 (+ 13us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260429 14:04:12.147383 8532 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:12.146136 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:44521 (local address 127.2.197.193:40727)
0429 14:04:12.146190 (+ 54us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:12.146193 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:12.146204 (+ 11us) server_negotiation.cc:408] Connection header received
0429 14:04:12.146236 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:12.146237 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:12.146243 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:12.146341 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:12.147045 (+ 704us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:12.147047 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:12.147047 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:04:12.147052 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:12.147069 (+ 17us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:04:12.147075 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:12.147175 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:12.147260 (+ 85us) server_negotiation.cc:300] Negotiation successful
0429 14:04:12.147276 (+ 16us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260429 14:04:12.147863 8535 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:12.146227 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:51153 (local address 127.2.197.194:36605)
0429 14:04:12.146294 (+ 67us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:12.146296 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:12.146409 (+ 113us) server_negotiation.cc:408] Connection header received
0429 14:04:12.146480 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:12.146481 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:12.146485 (+ 4us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:12.146671 (+ 186us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:12.147346 (+ 675us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:12.147348 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:12.147348 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:04:12.147353 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:12.147368 (+ 15us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:04:12.147372 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:12.147436 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:12.147762 (+ 326us) server_negotiation.cc:300] Negotiation successful
0429 14:04:12.147778 (+ 16us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260429 14:04:12.147992 8545 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:12.146129 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:36605 (local address 127.2.197.195:51153)
0429 14:04:12.146333 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:12.146349 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:12.146416 (+ 67us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:12.146743 (+ 327us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:12.146744 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:12.146747 (+ 3us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:12.147235 (+ 488us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 14:04:12.147236 (+ 1us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:12.147243 (+ 7us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 14:04:12.147254 (+ 11us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:12.147466 (+ 212us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:12.147467 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:12.147468 (+ 1us) client_negotiation.cc:770] Sending connection context
0429 14:04:12.147729 (+ 261us) client_negotiation.cc:241] Negotiation successful
0429 14:04:12.147748 (+ 19us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":129,"threads_started":1}
I20260429 14:04:12.147957 8205 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92926f3d8bf345dc8d3cb784f171d63b" candidate_uuid: "3daab3f5ab2f49aa96c8f3a4926c6ccc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffd4fe4c799f4108a282f78c586c8773" is_pre_election: true
I20260429 14:04:12.148126 8205 raft_consensus.cc:2468] T 92926f3d8bf345dc8d3cb784f171d63b P ffd4fe4c799f4108a282f78c586c8773 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3daab3f5ab2f49aa96c8f3a4926c6ccc in term 0.
I20260429 14:04:12.148363 8410 leader_election.cc:304] T 92926f3d8bf345dc8d3cb784f171d63b P 3daab3f5ab2f49aa96c8f3a4926c6ccc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3daab3f5ab2f49aa96c8f3a4926c6ccc, ffd4fe4c799f4108a282f78c586c8773; no voters:
I20260429 14:04:12.148430 8345 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92926f3d8bf345dc8d3cb784f171d63b" candidate_uuid: "3daab3f5ab2f49aa96c8f3a4926c6ccc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27bb31df5fa54c49b22e75a37da4aae0" is_pre_election: true
I20260429 14:04:12.148514 8543 raft_consensus.cc:2804] T 92926f3d8bf345dc8d3cb784f171d63b P 3daab3f5ab2f49aa96c8f3a4926c6ccc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 14:04:12.148557 8543 raft_consensus.cc:493] T 92926f3d8bf345dc8d3cb784f171d63b P 3daab3f5ab2f49aa96c8f3a4926c6ccc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 14:04:12.148603 8543 raft_consensus.cc:3060] T 92926f3d8bf345dc8d3cb784f171d63b P 3daab3f5ab2f49aa96c8f3a4926c6ccc [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:12.148598 8345 raft_consensus.cc:2468] T 92926f3d8bf345dc8d3cb784f171d63b P 27bb31df5fa54c49b22e75a37da4aae0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3daab3f5ab2f49aa96c8f3a4926c6ccc in term 0.
I20260429 14:04:12.149361 8543 raft_consensus.cc:515] T 92926f3d8bf345dc8d3cb784f171d63b P 3daab3f5ab2f49aa96c8f3a4926c6ccc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd4fe4c799f4108a282f78c586c8773" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 40727 } } peers { permanent_uuid: "3daab3f5ab2f49aa96c8f3a4926c6ccc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35849 } } peers { permanent_uuid: "27bb31df5fa54c49b22e75a37da4aae0" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 36605 } }
I20260429 14:04:12.149520 8543 leader_election.cc:290] T 92926f3d8bf345dc8d3cb784f171d63b P 3daab3f5ab2f49aa96c8f3a4926c6ccc [CANDIDATE]: Term 1 election: Requested vote from peers ffd4fe4c799f4108a282f78c586c8773 (127.2.197.193:40727), 27bb31df5fa54c49b22e75a37da4aae0 (127.2.197.194:36605)
I20260429 14:04:12.149794 8205 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92926f3d8bf345dc8d3cb784f171d63b" candidate_uuid: "3daab3f5ab2f49aa96c8f3a4926c6ccc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffd4fe4c799f4108a282f78c586c8773"
I20260429 14:04:12.149888 8205 raft_consensus.cc:3060] T 92926f3d8bf345dc8d3cb784f171d63b P ffd4fe4c799f4108a282f78c586c8773 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:12.149892 8345 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92926f3d8bf345dc8d3cb784f171d63b" candidate_uuid: "3daab3f5ab2f49aa96c8f3a4926c6ccc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27bb31df5fa54c49b22e75a37da4aae0"
I20260429 14:04:12.150130 8345 raft_consensus.cc:3060] T 92926f3d8bf345dc8d3cb784f171d63b P 27bb31df5fa54c49b22e75a37da4aae0 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:12.150723 8205 raft_consensus.cc:2468] T 92926f3d8bf345dc8d3cb784f171d63b P ffd4fe4c799f4108a282f78c586c8773 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3daab3f5ab2f49aa96c8f3a4926c6ccc in term 1.
I20260429 14:04:12.150945 8410 leader_election.cc:304] T 92926f3d8bf345dc8d3cb784f171d63b P 3daab3f5ab2f49aa96c8f3a4926c6ccc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3daab3f5ab2f49aa96c8f3a4926c6ccc, ffd4fe4c799f4108a282f78c586c8773; no voters:
I20260429 14:04:12.151042 8543 raft_consensus.cc:2804] T 92926f3d8bf345dc8d3cb784f171d63b P 3daab3f5ab2f49aa96c8f3a4926c6ccc [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:04:12.151211 8543 raft_consensus.cc:697] T 92926f3d8bf345dc8d3cb784f171d63b P 3daab3f5ab2f49aa96c8f3a4926c6ccc [term 1 LEADER]: Becoming Leader. State: Replica: 3daab3f5ab2f49aa96c8f3a4926c6ccc, State: Running, Role: LEADER
I20260429 14:04:12.151316 8543 consensus_queue.cc:237] T 92926f3d8bf345dc8d3cb784f171d63b P 3daab3f5ab2f49aa96c8f3a4926c6ccc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd4fe4c799f4108a282f78c586c8773" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 40727 } } peers { permanent_uuid: "3daab3f5ab2f49aa96c8f3a4926c6ccc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35849 } } peers { permanent_uuid: "27bb31df5fa54c49b22e75a37da4aae0" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 36605 } }
I20260429 14:04:12.151834 8345 raft_consensus.cc:2468] T 92926f3d8bf345dc8d3cb784f171d63b P 27bb31df5fa54c49b22e75a37da4aae0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3daab3f5ab2f49aa96c8f3a4926c6ccc in term 1.
I20260429 14:04:12.152196 8072 catalog_manager.cc:5671] T 92926f3d8bf345dc8d3cb784f171d63b P 3daab3f5ab2f49aa96c8f3a4926c6ccc reported cstate change: term changed from 0 to 1, leader changed from <none> to 3daab3f5ab2f49aa96c8f3a4926c6ccc (127.2.197.195). New cstate: current_term: 1 leader_uuid: "3daab3f5ab2f49aa96c8f3a4926c6ccc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd4fe4c799f4108a282f78c586c8773" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 40727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3daab3f5ab2f49aa96c8f3a4926c6ccc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27bb31df5fa54c49b22e75a37da4aae0" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 36605 } health_report { overall_health: UNKNOWN } } }
I20260429 14:04:12.161466 8533 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:12.160197 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55454 (local address 127.2.197.195:35849)
0429 14:04:12.160277 (+ 80us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:12.160280 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:12.160291 (+ 11us) server_negotiation.cc:408] Connection header received
0429 14:04:12.160349 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:12.160350 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:12.160360 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:12.160418 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:12.161147 (+ 729us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:12.161149 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:12.161149 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:04:12.161153 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:12.161168 (+ 15us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:04:12.161171 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:12.161228 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:12.161297 (+ 69us) server_negotiation.cc:300] Negotiation successful
0429 14:04:12.161347 (+ 50us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260429 14:04:12.163551 8345 raft_consensus.cc:1275] T 92926f3d8bf345dc8d3cb784f171d63b P 27bb31df5fa54c49b22e75a37da4aae0 [term 1 FOLLOWER]: Refusing update from remote peer 3daab3f5ab2f49aa96c8f3a4926c6ccc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:04:12.163650 8205 raft_consensus.cc:1275] T 92926f3d8bf345dc8d3cb784f171d63b P ffd4fe4c799f4108a282f78c586c8773 [term 1 FOLLOWER]: Refusing update from remote peer 3daab3f5ab2f49aa96c8f3a4926c6ccc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:04:12.163983 8546 consensus_queue.cc:1048] T 92926f3d8bf345dc8d3cb784f171d63b P 3daab3f5ab2f49aa96c8f3a4926c6ccc [LEADER]: Connected to new peer: Peer: permanent_uuid: "27bb31df5fa54c49b22e75a37da4aae0" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 36605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:04:12.164108 8543 consensus_queue.cc:1048] T 92926f3d8bf345dc8d3cb784f171d63b P 3daab3f5ab2f49aa96c8f3a4926c6ccc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffd4fe4c799f4108a282f78c586c8773" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 40727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:04:12.166649 8549 mvcc.cc:204] Tried to move back new op lower bound from 7280523068058394624 to 7280523068012560384. Current Snapshot: MvccSnapshot[applied={T|T < 7280523068058394624}]
I20260429 14:04:12.166676 8550 mvcc.cc:204] Tried to move back new op lower bound from 7280523068058394624 to 7280523068012560384. Current Snapshot: MvccSnapshot[applied={T|T < 7280523068058394624}]
I20260429 14:04:12.168103 8552 mvcc.cc:204] Tried to move back new op lower bound from 7280523068058394624 to 7280523068012560384. Current Snapshot: MvccSnapshot[applied={T|T < 7280523068058394624 or (T in {7280523068058394624})}]
I20260429 14:04:12.169224 8072 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:04:12.169307 8072 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:04:12.170195 8072 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 506c4acc0faf42c99c3aef713a534de4: Sending DeleteTablet for 3 replicas of tablet 92926f3d8bf345dc8d3cb784f171d63b
I20260429 14:04:12.170599 8325 tablet_service.cc:1558] Processing DeleteTablet for tablet 92926f3d8bf345dc8d3cb784f171d63b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:04:12 UTC) from {username='slave'} at 127.0.0.1:40684
I20260429 14:04:12.170611 8194 tablet_service.cc:1558] Processing DeleteTablet for tablet 92926f3d8bf345dc8d3cb784f171d63b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:04:12 UTC) from {username='slave'} at 127.0.0.1:60686
I20260429 14:04:12.170624 8456 tablet_service.cc:1558] Processing DeleteTablet for tablet 92926f3d8bf345dc8d3cb784f171d63b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:04:12 UTC) from {username='slave'} at 127.0.0.1:55442
I20260429 14:04:12.170836 8559 tablet_replica.cc:333] T 92926f3d8bf345dc8d3cb784f171d63b P 3daab3f5ab2f49aa96c8f3a4926c6ccc: stopping tablet replica
I20260429 14:04:12.170907 8559 raft_consensus.cc:2243] T 92926f3d8bf345dc8d3cb784f171d63b P 3daab3f5ab2f49aa96c8f3a4926c6ccc [term 1 LEADER]: Raft consensus shutting down.
I20260429 14:04:12.171005 8559 raft_consensus.cc:2272] T 92926f3d8bf345dc8d3cb784f171d63b P 3daab3f5ab2f49aa96c8f3a4926c6ccc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:04:12.171156 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 8132
I20260429 14:04:12.172412 8561 tablet_replica.cc:333] T 92926f3d8bf345dc8d3cb784f171d63b P 27bb31df5fa54c49b22e75a37da4aae0: stopping tablet replica
I20260429 14:04:12.172520 8561 raft_consensus.cc:2243] T 92926f3d8bf345dc8d3cb784f171d63b P 27bb31df5fa54c49b22e75a37da4aae0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:04:12.172655 8561 raft_consensus.cc:2272] T 92926f3d8bf345dc8d3cb784f171d63b P 27bb31df5fa54c49b22e75a37da4aae0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:04:12.174975 8559 ts_tablet_manager.cc:1916] T 92926f3d8bf345dc8d3cb784f171d63b P 3daab3f5ab2f49aa96c8f3a4926c6ccc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:04:12.176764 8559 ts_tablet_manager.cc:1929] T 92926f3d8bf345dc8d3cb784f171d63b P 3daab3f5ab2f49aa96c8f3a4926c6ccc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 14:04:12.176865 8559 log.cc:1199] T 92926f3d8bf345dc8d3cb784f171d63b P 3daab3f5ab2f49aa96c8f3a4926c6ccc: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777471413505459-2839-0/minicluster-data/ts-2/wal/wals/92926f3d8bf345dc8d3cb784f171d63b
I20260429 14:04:12.177125 8559 ts_tablet_manager.cc:1950] T 92926f3d8bf345dc8d3cb784f171d63b P 3daab3f5ab2f49aa96c8f3a4926c6ccc: Deleting consensus metadata
I20260429 14:04:12.177690 8059 catalog_manager.cc:5002] TS 3daab3f5ab2f49aa96c8f3a4926c6ccc (127.2.197.195:35849): tablet 92926f3d8bf345dc8d3cb784f171d63b (table test-table [id=ec7156f5c0a2423ca4075f7dfa952e4e]) successfully deleted
I20260429 14:04:12.178246 8561 ts_tablet_manager.cc:1916] T 92926f3d8bf345dc8d3cb784f171d63b P 27bb31df5fa54c49b22e75a37da4aae0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260429 14:04:12.178875 8057 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.197.193:40727 (error 108)
I20260429 14:04:12.179349 8530 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:12.179230 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:40727 (local address 127.0.0.1:60690)
0429 14:04:12.179280 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:12.179311 (+ 31us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.193:40727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260429 14:04:12.179458 8057 catalog_manager.cc:4729] TS ffd4fe4c799f4108a282f78c586c8773 (127.2.197.193:40727): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 92926f3d8bf345dc8d3cb784f171d63b: Network error: Client connection negotiation failed: client connection to 127.2.197.193:40727: connect: Connection refused (error 111)
I20260429 14:04:12.179908 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 8263
I20260429 14:04:12.185549 8530 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:12.185399 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:36605 (local address 127.0.0.1:40686)
0429 14:04:12.185466 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:12.185504 (+ 38us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.194:36605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260429 14:04:12.185689 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 8394
I20260429 14:04:12.190656 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 8041
2026-04-29T14:04:12Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (665 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260429 14:04:12.211126 2839 mini_oidc.cc:150] Starting JWKS server
I20260429 14:04:12.212139 2839 webserver.cc:492] Webserver started at https://127.0.0.1:42545/ using document root <none> and password file <none>
I20260429 14:04:12.212245 2839 mini_oidc.cc:186] Starting OIDC Discovery server
I20260429 14:04:12.212401 2839 webserver.cc:492] Webserver started at http://127.0.0.1:41935/ using document root <none> and password file <none>
2026-04-29T14:04:12Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:04:12Z Disabled control of system clock
I20260429 14:04:12.219514 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:40783
--webserver_interface=127.2.197.254
--webserver_port=0
--builtin_ntp_servers=127.2.197.212:43103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:40783
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42545/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777471413505459-2839-0/testchainca.cert with env {}
W20260429 14:04:12.299880 8568 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:12.300058 8568 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:12.300081 8568 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:12.301513 8568 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:04:12.301568 8568 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:12.301586 8568 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:12.301599 8568 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:04:12.301611 8568 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
W20260429 14:04:12.301625 8568 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
W20260429 14:04:12.301635 8568 flags.cc:432] Enabled experimental flag: --jwks_url=https://localhost:42545/jwks.json
I20260429 14:04:12.303463 8568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:43103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:40783
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:40783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42545/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777471413505459-2839-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:12.303676 8568 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:12.303817 8568 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:12.306088 8574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:12.306143 8576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:12.306143 8573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:12.306479 8568 server_base.cc:1061] running on GCE node
I20260429 14:04:12.306721 8568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:12.306968 8568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:12.308135 8568 hybrid_clock.cc:648] HybridClock initialized: now 1777471452308117 us; error 27 us; skew 500 ppm
I20260429 14:04:12.309232 8568 webserver.cc:492] Webserver started at http://127.2.197.254:41909/ using document root <none> and password file <none>
I20260429 14:04:12.309445 8568 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:12.309511 8568 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:12.309640 8568 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:04:12.310653 8568 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "22f66c7bb7d44758ad83a13b335e26be"
format_stamp: "Formatted at 2026-04-29 14:04:12 on dist-test-slave-8n77"
server_key: "ed5ad5ed610a52e44e4436f41f454444"
server_key_iv: "f476a03d2b1513fe6f85b2cafba6426e"
server_key_version: "encryptionkey@0"
I20260429 14:04:12.310979 8568 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "22f66c7bb7d44758ad83a13b335e26be"
format_stamp: "Formatted at 2026-04-29 14:04:12 on dist-test-slave-8n77"
server_key: "ed5ad5ed610a52e44e4436f41f454444"
server_key_iv: "f476a03d2b1513fe6f85b2cafba6426e"
server_key_version: "encryptionkey@0"
I20260429 14:04:12.312141 8568 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260429 14:04:12.312856 8582 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:12.313041 8568 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:12.313107 8568 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "22f66c7bb7d44758ad83a13b335e26be"
format_stamp: "Formatted at 2026-04-29 14:04:12 on dist-test-slave-8n77"
server_key: "ed5ad5ed610a52e44e4436f41f454444"
server_key_iv: "f476a03d2b1513fe6f85b2cafba6426e"
server_key_version: "encryptionkey@0"
I20260429 14:04:12.313153 8568 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:12.321352 8568 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:12.321619 8568 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:12.321714 8568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:12.325793 8568 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:40783
I20260429 14:04:12.325824 8634 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:40783 every 8 connection(s)
I20260429 14:04:12.326131 8568 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:04:12.326735 8635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:12.329072 8635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22f66c7bb7d44758ad83a13b335e26be: Bootstrap starting.
I20260429 14:04:12.329653 8635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22f66c7bb7d44758ad83a13b335e26be: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:12.329917 8635 log.cc:826] T 00000000000000000000000000000000 P 22f66c7bb7d44758ad83a13b335e26be: Log is configured to *not* fsync() on all Append() calls
I20260429 14:04:12.330664 8635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22f66c7bb7d44758ad83a13b335e26be: No bootstrap required, opened a new log
I20260429 14:04:12.331869 8635 raft_consensus.cc:359] T 00000000000000000000000000000000 P 22f66c7bb7d44758ad83a13b335e26be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f66c7bb7d44758ad83a13b335e26be" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40783 } }
I20260429 14:04:12.332008 8635 raft_consensus.cc:385] T 00000000000000000000000000000000 P 22f66c7bb7d44758ad83a13b335e26be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:12.332046 8635 raft_consensus.cc:740] T 00000000000000000000000000000000 P 22f66c7bb7d44758ad83a13b335e26be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22f66c7bb7d44758ad83a13b335e26be, State: Initialized, Role: FOLLOWER
I20260429 14:04:12.332201 8635 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22f66c7bb7d44758ad83a13b335e26be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f66c7bb7d44758ad83a13b335e26be" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40783 } }
I20260429 14:04:12.332297 8635 raft_consensus.cc:399] T 00000000000000000000000000000000 P 22f66c7bb7d44758ad83a13b335e26be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:04:12.332335 8635 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22f66c7bb7d44758ad83a13b335e26be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:04:12.332381 8635 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 22f66c7bb7d44758ad83a13b335e26be [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:12.332914 8635 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22f66c7bb7d44758ad83a13b335e26be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f66c7bb7d44758ad83a13b335e26be" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40783 } }
I20260429 14:04:12.333039 8635 leader_election.cc:304] T 00000000000000000000000000000000 P 22f66c7bb7d44758ad83a13b335e26be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22f66c7bb7d44758ad83a13b335e26be; no voters:
I20260429 14:04:12.333205 8635 leader_election.cc:290] T 00000000000000000000000000000000 P 22f66c7bb7d44758ad83a13b335e26be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:04:12.333328 8638 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 22f66c7bb7d44758ad83a13b335e26be [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:04:12.333454 8635 sys_catalog.cc:565] T 00000000000000000000000000000000 P 22f66c7bb7d44758ad83a13b335e26be [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:04:12.333511 8638 raft_consensus.cc:697] T 00000000000000000000000000000000 P 22f66c7bb7d44758ad83a13b335e26be [term 1 LEADER]: Becoming Leader. State: Replica: 22f66c7bb7d44758ad83a13b335e26be, State: Running, Role: LEADER
I20260429 14:04:12.333606 8638 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22f66c7bb7d44758ad83a13b335e26be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f66c7bb7d44758ad83a13b335e26be" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40783 } }
I20260429 14:04:12.334009 8640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22f66c7bb7d44758ad83a13b335e26be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22f66c7bb7d44758ad83a13b335e26be. Latest consensus state: current_term: 1 leader_uuid: "22f66c7bb7d44758ad83a13b335e26be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f66c7bb7d44758ad83a13b335e26be" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40783 } } }
I20260429 14:04:12.334113 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 8568
I20260429 14:04:12.334144 8640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22f66c7bb7d44758ad83a13b335e26be [sys.catalog]: This master's current role is: LEADER
I20260429 14:04:12.334227 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:04:12.333989 8639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22f66c7bb7d44758ad83a13b335e26be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22f66c7bb7d44758ad83a13b335e26be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f66c7bb7d44758ad83a13b335e26be" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40783 } } }
I20260429 14:04:12.334331 8639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22f66c7bb7d44758ad83a13b335e26be [sys.catalog]: This master's current role is: LEADER
I20260429 14:04:12.334434 2839 external_mini_cluster.cc:1468] Setting key c770ffc74b2078ce646e1cde356f6e6e
I20260429 14:04:12.334710 8647 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:04:12.335176 8647 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:04:12.337765 8647 catalog_manager.cc:1357] Generated new cluster ID: 093f4d9245ca4eb586a3d7650a3a16a4
I20260429 14:04:12.337822 8647 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:04:12.339391 8656 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:12.335058 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48006 (local address 127.2.197.254:40783)
0429 14:04:12.335424 (+ 366us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:12.335431 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:12.335454 (+ 23us) server_negotiation.cc:408] Connection header received
0429 14:04:12.335556 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:12.335563 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:12.335617 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:12.335742 (+ 125us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:12.336260 (+ 518us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:12.337348 (+ 1088us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:12.338056 (+ 708us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:12.338314 (+ 258us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:12.338794 (+ 480us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:12.338806 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:12.338807 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:04:12.339033 (+ 226us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:12.339054 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:04:12.339062 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:12.339136 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:12.339206 (+ 70us) server_negotiation.cc:300] Negotiation successful
0429 14:04:12.339251 (+ 45us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":67,"threads_started":1}
I20260429 14:04:12.340293 2839 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260429 14:04:12.349651 8647 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:04:12.350194 8647 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:04:12.356063 8647 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 22f66c7bb7d44758ad83a13b335e26be: Generated new TSK 0
I20260429 14:04:12.356279 8647 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 14:04:12.393642 8656 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:12.366188 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48014 (local address 127.2.197.254:40783)
0429 14:04:12.366243 (+ 55us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:12.366245 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:12.366380 (+ 135us) server_negotiation.cc:408] Connection header received
0429 14:04:12.366478 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:12.366479 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:12.366491 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:12.366681 (+ 190us) server_negotiation.cc:227] Negotiated authn=JWT
0429 14:04:12.367107 (+ 426us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:12.367702 (+ 595us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:12.368611 (+ 909us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:12.368758 (+ 147us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:12.368801 (+ 43us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0429 14:04:12.393248 (+ 24447us) server_negotiation.cc:378] Sending JWT_EXCHANGE NegotiatePB response
0429 14:04:12.393314 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:12.393444 (+ 130us) server_negotiation.cc:300] Negotiation successful
0429 14:04:12.393534 (+ 90us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25,"thread_start_us":155,"threads_started":1}
I20260429 14:04:12.402153 8656 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:12.399291 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48022 (local address 127.2.197.254:40783)
0429 14:04:12.399362 (+ 71us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:12.399365 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:12.399421 (+ 56us) server_negotiation.cc:408] Connection header received
0429 14:04:12.399474 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:12.399476 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:12.399493 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:12.399557 (+ 64us) server_negotiation.cc:227] Negotiated authn=JWT
0429 14:04:12.399823 (+ 266us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:12.400645 (+ 822us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:12.401267 (+ 622us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:12.401460 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:12.401493 (+ 33us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0429 14:04:12.401951 (+ 458us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0429 14:04:12.402087 (+ 136us) negotiation.cc:326] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48022: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"server-negotiator.queue_time_us":44}
W20260429 14:04:12.402231 8656 negotiation.cc:343] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48022: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
W20260429 14:04:12.402429 8671 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:12.399148 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40783 (local address 127.0.0.1:48022)
0429 14:04:12.399347 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:12.399357 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:12.399425 (+ 68us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:12.399570 (+ 145us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:12.399570 (+ 0us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:12.399572 (+ 2us) client_negotiation.cc:190] Negotiated authn=JWT
0429 14:04:12.399754 (+ 182us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:12.399756 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:12.400747 (+ 991us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:12.400748 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:12.401157 (+ 409us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:12.401159 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:12.401222 (+ 63us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:12.401226 (+ 4us) client_negotiation.cc:253] Sending JWT_EXCHANGE NegotiatePB request
0429 14:04:12.402071 (+ 845us) client_negotiation.cc:284] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0429 14:04:12.402162 (+ 91us) negotiation.cc:326] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.197.254:40783: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":107,"threads_started":1}
I20260429 14:04:27.408316 8677 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:27.405869 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53078 (local address 127.2.197.254:40783)
0429 14:04:27.406019 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:27.406021 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:27.406064 (+ 43us) server_negotiation.cc:408] Connection header received
0429 14:04:27.406209 (+ 145us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:27.406210 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:27.406221 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:27.406382 (+ 161us) server_negotiation.cc:227] Negotiated authn=JWT
0429 14:04:27.406652 (+ 270us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:27.407171 (+ 519us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:27.407825 (+ 654us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:27.407961 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:27.407980 (+ 19us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0429 14:04:27.408143 (+ 163us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0429 14:04:27.408264 (+ 121us) negotiation.cc:326] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53078: JWT verification failed: token expired
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
W20260429 14:04:27.408416 8677 negotiation.cc:343] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53078: JWT verification failed: token expired
W20260429 14:04:27.408440 8676 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:27.405683 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40783 (local address 127.0.0.1:53078)
0429 14:04:27.405995 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:27.406007 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:27.406078 (+ 71us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:27.406384 (+ 306us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:27.406384 (+ 0us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:27.406386 (+ 2us) client_negotiation.cc:190] Negotiated authn=JWT
0429 14:04:27.406558 (+ 172us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:27.406559 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:27.407275 (+ 716us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:27.407276 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:27.407706 (+ 430us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:27.407707 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:27.407771 (+ 64us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:27.407776 (+ 5us) client_negotiation.cc:253] Sending JWT_EXCHANGE NegotiatePB request
0429 14:04:27.408259 (+ 483us) client_negotiation.cc:284] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0429 14:04:27.408366 (+ 107us) negotiation.cc:326] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.197.254:40783: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":247,"threads_started":1}
W20260429 14:04:27.410233 8677 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:27.409698 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53084 (local address 127.2.197.254:40783)
0429 14:04:27.409754 (+ 56us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:27.409756 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:27.409879 (+ 123us) server_negotiation.cc:408] Connection header received
0429 14:04:27.409911 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:27.409912 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:27.409923 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:27.410001 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:27.410186 (+ 185us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53084: BlockingRecv error: recv got EOF from 127.0.0.1:53084 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260429 14:04:27.413148 8687 client_negotiation.cc:376] the client has a JWT but it isn't advertising its JWT-based authn capability since it doesn't trust the certificate of the server at 127.2.197.254:40783
W20260429 14:04:27.413513 8677 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:27.413010 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53088 (local address 127.2.197.254:40783)
0429 14:04:27.413068 (+ 58us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:27.413070 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:27.413242 (+ 172us) server_negotiation.cc:408] Connection header received
0429 14:04:27.413275 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:27.413277 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:27.413289 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:27.413359 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:27.413469 (+ 110us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53088: BlockingRecv error: recv got EOF from 127.0.0.1:53088 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260429 14:04:27.413672 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 8568
2026-04-29T14:04:27Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniCluster (15241 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260429 14:04:27.452850 2839 mini_oidc.cc:150] Starting JWKS server
I20260429 14:04:27.453625 2839 webserver.cc:492] Webserver started at https://127.0.0.1:41265/ using document root <none> and password file <none>
I20260429 14:04:27.453698 2839 mini_oidc.cc:186] Starting OIDC Discovery server
I20260429 14:04:27.453860 2839 webserver.cc:492] Webserver started at http://127.0.0.1:35093/ using document root <none> and password file <none>
2026-04-29T14:04:27Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:04:27Z Disabled control of system clock
I20260429 14:04:27.461078 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:44281
--webserver_interface=127.2.197.254
--webserver_port=0
--builtin_ntp_servers=127.2.197.212:41305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:44281
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41265/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777471413505459-2839-0/testchainca.cert with env {}
W20260429 14:04:27.540136 8693 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:27.540325 8693 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:27.540349 8693 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:27.541786 8693 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:04:27.541841 8693 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:27.541854 8693 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:27.541867 8693 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:04:27.541878 8693 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
W20260429 14:04:27.541891 8693 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
W20260429 14:04:27.541903 8693 flags.cc:432] Enabled experimental flag: --jwks_url=https://localhost:41265/jwks.json
I20260429 14:04:27.543566 8693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:41305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:44281
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:44281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41265/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777471413505459-2839-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:27.543767 8693 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:27.543903 8693 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:27.546365 8699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:27.546459 8693 server_base.cc:1061] running on GCE node
W20260429 14:04:27.546365 8701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:27.546365 8698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:27.546837 8693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:27.547086 8693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:27.548265 8693 hybrid_clock.cc:648] HybridClock initialized: now 1777471467548239 us; error 34 us; skew 500 ppm
I20260429 14:04:27.549607 8693 webserver.cc:492] Webserver started at http://127.2.197.254:36497/ using document root <none> and password file <none>
I20260429 14:04:27.549819 8693 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:27.549886 8693 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:27.549991 8693 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:04:27.550973 8693 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "2cf1b8ebf9074c20985c5517d31a52b7"
format_stamp: "Formatted at 2026-04-29 14:04:27 on dist-test-slave-8n77"
server_key: "5ac150c67f39899ba3bb8acf01a49341"
server_key_iv: "573ce906600a686ce4c518db389ea5aa"
server_key_version: "encryptionkey@0"
I20260429 14:04:27.551306 8693 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "2cf1b8ebf9074c20985c5517d31a52b7"
format_stamp: "Formatted at 2026-04-29 14:04:27 on dist-test-slave-8n77"
server_key: "5ac150c67f39899ba3bb8acf01a49341"
server_key_iv: "573ce906600a686ce4c518db389ea5aa"
server_key_version: "encryptionkey@0"
I20260429 14:04:27.552541 8693 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260429 14:04:27.553337 8707 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:27.553668 8693 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:27.553755 8693 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "2cf1b8ebf9074c20985c5517d31a52b7"
format_stamp: "Formatted at 2026-04-29 14:04:27 on dist-test-slave-8n77"
server_key: "5ac150c67f39899ba3bb8acf01a49341"
server_key_iv: "573ce906600a686ce4c518db389ea5aa"
server_key_version: "encryptionkey@0"
I20260429 14:04:27.553846 8693 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:27.568305 8693 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:27.568657 8693 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:27.568804 8693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:27.572813 8693 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:44281
I20260429 14:04:27.572875 8759 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:44281 every 8 connection(s)
I20260429 14:04:27.573205 8693 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:04:27.573832 8760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:27.575348 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 8693
I20260429 14:04:27.575448 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:04:27.575616 2839 external_mini_cluster.cc:1468] Setting key 70eb7aec5513a3b18991a0e52b8eb96b
I20260429 14:04:27.577014 8760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cf1b8ebf9074c20985c5517d31a52b7: Bootstrap starting.
I20260429 14:04:27.577749 8760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cf1b8ebf9074c20985c5517d31a52b7: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:27.578109 8760 log.cc:826] T 00000000000000000000000000000000 P 2cf1b8ebf9074c20985c5517d31a52b7: Log is configured to *not* fsync() on all Append() calls
I20260429 14:04:27.578822 8760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cf1b8ebf9074c20985c5517d31a52b7: No bootstrap required, opened a new log
I20260429 14:04:27.580143 8760 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2cf1b8ebf9074c20985c5517d31a52b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf1b8ebf9074c20985c5517d31a52b7" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 44281 } }
I20260429 14:04:27.580291 8760 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2cf1b8ebf9074c20985c5517d31a52b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:27.580343 8760 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2cf1b8ebf9074c20985c5517d31a52b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cf1b8ebf9074c20985c5517d31a52b7, State: Initialized, Role: FOLLOWER
I20260429 14:04:27.580493 8760 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cf1b8ebf9074c20985c5517d31a52b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf1b8ebf9074c20985c5517d31a52b7" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 44281 } }
I20260429 14:04:27.580602 8760 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2cf1b8ebf9074c20985c5517d31a52b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:04:27.580649 8760 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2cf1b8ebf9074c20985c5517d31a52b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:04:27.580701 8760 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2cf1b8ebf9074c20985c5517d31a52b7 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:27.581288 8760 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2cf1b8ebf9074c20985c5517d31a52b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf1b8ebf9074c20985c5517d31a52b7" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 44281 } }
I20260429 14:04:27.581422 8760 leader_election.cc:304] T 00000000000000000000000000000000 P 2cf1b8ebf9074c20985c5517d31a52b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cf1b8ebf9074c20985c5517d31a52b7; no voters:
I20260429 14:04:27.581648 8760 leader_election.cc:290] T 00000000000000000000000000000000 P 2cf1b8ebf9074c20985c5517d31a52b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:04:27.581686 8765 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2cf1b8ebf9074c20985c5517d31a52b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:04:27.581800 8765 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2cf1b8ebf9074c20985c5517d31a52b7 [term 1 LEADER]: Becoming Leader. State: Replica: 2cf1b8ebf9074c20985c5517d31a52b7, State: Running, Role: LEADER
I20260429 14:04:27.581881 8765 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cf1b8ebf9074c20985c5517d31a52b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf1b8ebf9074c20985c5517d31a52b7" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 44281 } }
I20260429 14:04:27.581908 8760 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2cf1b8ebf9074c20985c5517d31a52b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:04:27.582996 8767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cf1b8ebf9074c20985c5517d31a52b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cf1b8ebf9074c20985c5517d31a52b7. Latest consensus state: current_term: 1 leader_uuid: "2cf1b8ebf9074c20985c5517d31a52b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf1b8ebf9074c20985c5517d31a52b7" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 44281 } } }
I20260429 14:04:27.583101 8767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cf1b8ebf9074c20985c5517d31a52b7 [sys.catalog]: This master's current role is: LEADER
I20260429 14:04:27.583211 8766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cf1b8ebf9074c20985c5517d31a52b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2cf1b8ebf9074c20985c5517d31a52b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf1b8ebf9074c20985c5517d31a52b7" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 44281 } } }
I20260429 14:04:27.583309 8766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cf1b8ebf9074c20985c5517d31a52b7 [sys.catalog]: This master's current role is: LEADER
I20260429 14:04:27.583623 8775 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:04:27.584532 8763 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:27.576213 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60594 (local address 127.2.197.254:44281)
0429 14:04:27.576417 (+ 204us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:27.576424 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:27.576450 (+ 26us) server_negotiation.cc:408] Connection header received
0429 14:04:27.576871 (+ 421us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:27.576880 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:27.576942 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:27.577110 (+ 168us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:27.577605 (+ 495us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:27.578526 (+ 921us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:27.579163 (+ 637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:27.579420 (+ 257us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:27.583831 (+ 4411us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:27.583843 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:27.583843 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:04:27.584009 (+ 166us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:27.584028 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:04:27.584034 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:27.584091 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:27.584329 (+ 238us) server_negotiation.cc:300] Negotiation successful
0429 14:04:27.584391 (+ 62us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":77,"threads_started":1}
I20260429 14:04:27.584657 8775 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:04:27.586150 2839 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260429 14:04:27.586395 8775 catalog_manager.cc:1357] Generated new cluster ID: 2a4e13717009450b94cd944c47cb4363
I20260429 14:04:27.586447 8775 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:04:27.602746 8775 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:04:27.603307 8775 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:04:27.608808 8775 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 2cf1b8ebf9074c20985c5517d31a52b7: Generated new TSK 0
I20260429 14:04:27.608974 8775 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20260429 14:04:27.641624 8790 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260429 14:04:27.641719 8790 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260429 14:04:27.642045 8763 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:27.618659 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60600 (local address 127.2.197.254:44281)
0429 14:04:27.618712 (+ 53us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:27.618715 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:27.618833 (+ 118us) server_negotiation.cc:408] Connection header received
0429 14:04:27.618921 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:27.618922 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:27.618934 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:27.618991 (+ 57us) server_negotiation.cc:227] Negotiated authn=JWT
0429 14:04:27.619339 (+ 348us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:27.620283 (+ 944us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:27.621186 (+ 903us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:27.621330 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:27.621372 (+ 42us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0429 14:04:27.641784 (+ 20412us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41265/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0429 14:04:27.641990 (+ 206us) negotiation.cc:326] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60600: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41265/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":27}
W20260429 14:04:27.642125 8763 negotiation.cc:343] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60600: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41265/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260429 14:04:27.642165 8789 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:27.618558 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:44281 (local address 127.0.0.1:60600)
0429 14:04:27.618741 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:27.618754 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:27.618836 (+ 82us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:27.619013 (+ 177us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:27.619014 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:27.619016 (+ 2us) client_negotiation.cc:190] Negotiated authn=JWT
0429 14:04:27.619263 (+ 247us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:27.619265 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:27.620383 (+ 1118us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:27.620385 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:27.621046 (+ 661us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:27.621049 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:27.621145 (+ 96us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:27.621151 (+ 6us) client_negotiation.cc:253] Sending JWT_EXCHANGE NegotiatePB request
0429 14:04:27.641955 (+ 20804us) client_negotiation.cc:284] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41265/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0429 14:04:27.642076 (+ 121us) negotiation.cc:326] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.197.254:44281: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41265/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":107,"threads_started":1}
I20260429 14:04:27.642517 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 8693
2026-04-29T14:04:27Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (227 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260429 14:04:27.680748 2839 mini_oidc.cc:150] Starting JWKS server
I20260429 14:04:27.681496 2839 webserver.cc:492] Webserver started at https://127.0.0.1:44951/ using document root <none> and password file <none>
I20260429 14:04:27.681537 2839 mini_oidc.cc:186] Starting OIDC Discovery server
I20260429 14:04:27.681663 2839 webserver.cc:492] Webserver started at http://127.0.0.1:37761/ using document root <none> and password file <none>
2026-04-29T14:04:27Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:04:27Z Disabled control of system clock
I20260429 14:04:27.688647 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:38563
--webserver_interface=127.2.197.254
--webserver_port=0
--builtin_ntp_servers=127.2.197.212:45757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:38563
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44951/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260429 14:04:27.767549 8796 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:27.767781 8796 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:27.767817 8796 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:27.769345 8796 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:04:27.769429 8796 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:27.769449 8796 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:27.769469 8796 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:04:27.769496 8796 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
W20260429 14:04:27.769528 8796 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
W20260429 14:04:27.769560 8796 flags.cc:432] Enabled experimental flag: --jwks_url=https://localhost:44951/jwks.json
I20260429 14:04:27.771284 8796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:45757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:38563
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:38563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44951/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:27.771579 8796 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:27.771816 8796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:27.774582 8804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:27.774616 8802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:27.774582 8801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:27.775097 8796 server_base.cc:1061] running on GCE node
I20260429 14:04:27.775272 8796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:27.775471 8796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:27.776670 8796 hybrid_clock.cc:648] HybridClock initialized: now 1777471467776646 us; error 38 us; skew 500 ppm
I20260429 14:04:27.777828 8796 webserver.cc:492] Webserver started at http://127.2.197.254:41157/ using document root <none> and password file <none>
I20260429 14:04:27.778013 8796 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:27.778056 8796 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:27.778131 8796 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:04:27.779132 8796 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "1ffb73169e8a46da986695adbdfdbf51"
format_stamp: "Formatted at 2026-04-29 14:04:27 on dist-test-slave-8n77"
server_key: "05aa50b1831bdc57e53e3214b9f0d659"
server_key_iv: "66a4fa4c02a4a4428849510d796f05b3"
server_key_version: "encryptionkey@0"
I20260429 14:04:27.779407 8796 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "1ffb73169e8a46da986695adbdfdbf51"
format_stamp: "Formatted at 2026-04-29 14:04:27 on dist-test-slave-8n77"
server_key: "05aa50b1831bdc57e53e3214b9f0d659"
server_key_iv: "66a4fa4c02a4a4428849510d796f05b3"
server_key_version: "encryptionkey@0"
I20260429 14:04:27.780665 8796 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260429 14:04:27.781574 8810 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:27.781813 8796 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:27.781886 8796 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "1ffb73169e8a46da986695adbdfdbf51"
format_stamp: "Formatted at 2026-04-29 14:04:27 on dist-test-slave-8n77"
server_key: "05aa50b1831bdc57e53e3214b9f0d659"
server_key_iv: "66a4fa4c02a4a4428849510d796f05b3"
server_key_version: "encryptionkey@0"
I20260429 14:04:27.781949 8796 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:27.807711 8796 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:27.807998 8796 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:27.808095 8796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:27.812062 8796 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:38563
I20260429 14:04:27.812126 8862 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:38563 every 8 connection(s)
I20260429 14:04:27.812425 8796 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:04:27.812847 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 8796
I20260429 14:04:27.812922 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:04:27.813097 2839 external_mini_cluster.cc:1468] Setting key 2f807a9ba931f67dcf14183e93dafc73
I20260429 14:04:27.813103 8863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:27.816175 8863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ffb73169e8a46da986695adbdfdbf51: Bootstrap starting.
I20260429 14:04:27.816854 8863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ffb73169e8a46da986695adbdfdbf51: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:27.817183 8863 log.cc:826] T 00000000000000000000000000000000 P 1ffb73169e8a46da986695adbdfdbf51: Log is configured to *not* fsync() on all Append() calls
I20260429 14:04:27.818024 8863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ffb73169e8a46da986695adbdfdbf51: No bootstrap required, opened a new log
I20260429 14:04:27.818318 8866 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:27.813693 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44172 (local address 127.2.197.254:38563)
0429 14:04:27.813892 (+ 199us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:27.813897 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:27.813914 (+ 17us) server_negotiation.cc:408] Connection header received
0429 14:04:27.814791 (+ 877us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:27.814798 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:27.814849 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:27.815010 (+ 161us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:27.815511 (+ 501us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:27.816240 (+ 729us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:27.816856 (+ 616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:27.817113 (+ 257us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:27.817642 (+ 529us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:27.817649 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:27.817649 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:04:27.817901 (+ 252us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:27.817921 (+ 20us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:04:27.817930 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:27.818015 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:27.818087 (+ 72us) server_negotiation.cc:300] Negotiation successful
0429 14:04:27.818141 (+ 54us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":87,"threads_started":1}
I20260429 14:04:27.819439 2839 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260429 14:04:27.819657 8863 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1ffb73169e8a46da986695adbdfdbf51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ffb73169e8a46da986695adbdfdbf51" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 38563 } }
I20260429 14:04:27.819783 8863 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1ffb73169e8a46da986695adbdfdbf51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:27.819808 8863 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1ffb73169e8a46da986695adbdfdbf51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ffb73169e8a46da986695adbdfdbf51, State: Initialized, Role: FOLLOWER
I20260429 14:04:27.819958 8863 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ffb73169e8a46da986695adbdfdbf51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ffb73169e8a46da986695adbdfdbf51" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 38563 } }
I20260429 14:04:27.820027 8863 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1ffb73169e8a46da986695adbdfdbf51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:04:27.820053 8863 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1ffb73169e8a46da986695adbdfdbf51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:04:27.820122 8863 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1ffb73169e8a46da986695adbdfdbf51 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:27.820879 8863 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1ffb73169e8a46da986695adbdfdbf51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ffb73169e8a46da986695adbdfdbf51" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 38563 } }
I20260429 14:04:27.820976 8863 leader_election.cc:304] T 00000000000000000000000000000000 P 1ffb73169e8a46da986695adbdfdbf51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ffb73169e8a46da986695adbdfdbf51; no voters:
I20260429 14:04:27.821197 8863 leader_election.cc:290] T 00000000000000000000000000000000 P 1ffb73169e8a46da986695adbdfdbf51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:04:27.821296 8868 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1ffb73169e8a46da986695adbdfdbf51 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:04:27.821444 8863 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1ffb73169e8a46da986695adbdfdbf51 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:04:27.821455 8868 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1ffb73169e8a46da986695adbdfdbf51 [term 1 LEADER]: Becoming Leader. State: Replica: 1ffb73169e8a46da986695adbdfdbf51, State: Running, Role: LEADER
I20260429 14:04:27.821599 8868 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ffb73169e8a46da986695adbdfdbf51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ffb73169e8a46da986695adbdfdbf51" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 38563 } }
I20260429 14:04:27.821965 8869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ffb73169e8a46da986695adbdfdbf51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ffb73169e8a46da986695adbdfdbf51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ffb73169e8a46da986695adbdfdbf51" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 38563 } } }
I20260429 14:04:27.822044 8869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ffb73169e8a46da986695adbdfdbf51 [sys.catalog]: This master's current role is: LEADER
I20260429 14:04:27.822239 8870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ffb73169e8a46da986695adbdfdbf51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ffb73169e8a46da986695adbdfdbf51. Latest consensus state: current_term: 1 leader_uuid: "1ffb73169e8a46da986695adbdfdbf51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ffb73169e8a46da986695adbdfdbf51" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 38563 } } }
I20260429 14:04:27.822343 8870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ffb73169e8a46da986695adbdfdbf51 [sys.catalog]: This master's current role is: LEADER
I20260429 14:04:27.822299 8875 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:04:27.822906 8875 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:04:27.824448 8875 catalog_manager.cc:1357] Generated new cluster ID: f1b710bd71cb4fd69e88eddc560d13f2
I20260429 14:04:27.824496 8875 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:04:27.846439 8875 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:04:27.847050 8875 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:04:27.852098 8875 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 1ffb73169e8a46da986695adbdfdbf51: Generated new TSK 0
I20260429 14:04:27.852393 8875 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20260429 14:04:27.889626 8893 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260429 14:04:27.889716 8893 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260429 14:04:27.890805 8892 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:27.866815 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:38563 (local address 127.0.0.1:44180)
0429 14:04:27.866981 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:27.866994 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:27.867067 (+ 73us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:27.867254 (+ 187us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:27.867255 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:27.867256 (+ 1us) client_negotiation.cc:190] Negotiated authn=JWT
0429 14:04:27.867591 (+ 335us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:27.867595 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:27.868444 (+ 849us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:27.868445 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:27.868856 (+ 411us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:27.868858 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:27.868979 (+ 121us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:27.868985 (+ 6us) client_negotiation.cc:253] Sending JWT_EXCHANGE NegotiatePB request
0429 14:04:27.890612 (+ 21627us) client_negotiation.cc:284] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44951/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0429 14:04:27.890703 (+ 91us) negotiation.cc:326] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.197.254:38563: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44951/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":110,"threads_started":1}
I20260429 14:04:27.890820 8866 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:27.866952 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44180 (local address 127.2.197.254:38563)
0429 14:04:27.867004 (+ 52us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:27.867006 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:27.867047 (+ 41us) server_negotiation.cc:408] Connection header received
0429 14:04:27.867167 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:27.867168 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:27.867179 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:27.867250 (+ 71us) server_negotiation.cc:227] Negotiated authn=JWT
0429 14:04:27.867768 (+ 518us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:27.868337 (+ 569us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:27.868966 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:27.869139 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:27.869184 (+ 45us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0429 14:04:27.890504 (+ 21320us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44951/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0429 14:04:27.890761 (+ 257us) negotiation.cc:326] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44180: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44951/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":25}
W20260429 14:04:27.890872 8866 negotiation.cc:343] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44180: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44951/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260429 14:04:27.891181 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 8796
2026-04-29T14:04:27Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (248 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-04-29T14:04:27Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:04:27Z Disabled control of system clock
I20260429 14:04:27.936524 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:37963
--webserver_interface=127.2.197.254
--webserver_port=0
--builtin_ntp_servers=127.2.197.212:33611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:37963
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777471413505459-2839-0/testchainca.cert with env {}
W20260429 14:04:28.014678 8897 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:28.014930 8897 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:28.014969 8897 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:28.016541 8897 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:04:28.016633 8897 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:28.016654 8897 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:28.016673 8897 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:04:28.016700 8897 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
W20260429 14:04:28.016732 8897 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
W20260429 14:04:28.016755 8897 flags.cc:432] Enabled experimental flag: --jwks_url=default.url
I20260429 14:04:28.018571 8897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:33611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:37963
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:37963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777471413505459-2839-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:28.018870 8897 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:28.019073 8897 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:28.021509 8903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:28.021517 8902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:28.021503 8905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:28.022174 8897 server_base.cc:1061] running on GCE node
I20260429 14:04:28.022430 8897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:28.022768 8897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:28.023928 8897 hybrid_clock.cc:648] HybridClock initialized: now 1777471468023921 us; error 46 us; skew 500 ppm
I20260429 14:04:28.025743 8897 webserver.cc:492] Webserver started at http://127.2.197.254:44631/ using document root <none> and password file <none>
I20260429 14:04:28.026062 8897 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:28.026145 8897 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:28.026250 8897 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:04:28.027758 8897 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "3bd66da4382c45ffb3c5da2796583494"
format_stamp: "Formatted at 2026-04-29 14:04:28 on dist-test-slave-8n77"
server_key: "d4a9d345c822e106ca8fb6bcfb8fe29d"
server_key_iv: "46b17364875d8c33c085c52be08fdb74"
server_key_version: "encryptionkey@0"
I20260429 14:04:28.028147 8897 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "3bd66da4382c45ffb3c5da2796583494"
format_stamp: "Formatted at 2026-04-29 14:04:28 on dist-test-slave-8n77"
server_key: "d4a9d345c822e106ca8fb6bcfb8fe29d"
server_key_iv: "46b17364875d8c33c085c52be08fdb74"
server_key_version: "encryptionkey@0"
I20260429 14:04:28.029728 8897 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260429 14:04:28.030741 8911 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:28.030977 8897 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260429 14:04:28.031064 8897 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "3bd66da4382c45ffb3c5da2796583494"
format_stamp: "Formatted at 2026-04-29 14:04:28 on dist-test-slave-8n77"
server_key: "d4a9d345c822e106ca8fb6bcfb8fe29d"
server_key_iv: "46b17364875d8c33c085c52be08fdb74"
server_key_version: "encryptionkey@0"
I20260429 14:04:28.031147 8897 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:28.037730 8897 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:28.038033 8897 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:28.038165 8897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:28.042186 8897 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:37963
I20260429 14:04:28.042258 8963 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:37963 every 8 connection(s)
I20260429 14:04:28.042605 8897 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:04:28.043183 8964 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:28.045967 8964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bd66da4382c45ffb3c5da2796583494: Bootstrap starting.
I20260429 14:04:28.046631 8964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bd66da4382c45ffb3c5da2796583494: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:28.046851 8964 log.cc:826] T 00000000000000000000000000000000 P 3bd66da4382c45ffb3c5da2796583494: Log is configured to *not* fsync() on all Append() calls
I20260429 14:04:28.047550 8964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bd66da4382c45ffb3c5da2796583494: No bootstrap required, opened a new log
I20260429 14:04:28.048730 8964 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3bd66da4382c45ffb3c5da2796583494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd66da4382c45ffb3c5da2796583494" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 37963 } }
I20260429 14:04:28.048858 8964 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3bd66da4382c45ffb3c5da2796583494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:28.048880 8964 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3bd66da4382c45ffb3c5da2796583494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bd66da4382c45ffb3c5da2796583494, State: Initialized, Role: FOLLOWER
I20260429 14:04:28.049010 8964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bd66da4382c45ffb3c5da2796583494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd66da4382c45ffb3c5da2796583494" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 37963 } }
I20260429 14:04:28.049088 8964 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3bd66da4382c45ffb3c5da2796583494 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:04:28.049109 8964 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3bd66da4382c45ffb3c5da2796583494 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:04:28.049135 8964 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3bd66da4382c45ffb3c5da2796583494 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:28.049655 8964 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3bd66da4382c45ffb3c5da2796583494 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd66da4382c45ffb3c5da2796583494" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 37963 } }
I20260429 14:04:28.049750 8964 leader_election.cc:304] T 00000000000000000000000000000000 P 3bd66da4382c45ffb3c5da2796583494 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bd66da4382c45ffb3c5da2796583494; no voters:
I20260429 14:04:28.049876 8964 leader_election.cc:290] T 00000000000000000000000000000000 P 3bd66da4382c45ffb3c5da2796583494 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:04:28.049988 8967 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3bd66da4382c45ffb3c5da2796583494 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:04:28.050148 8964 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3bd66da4382c45ffb3c5da2796583494 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:04:28.050171 8967 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3bd66da4382c45ffb3c5da2796583494 [term 1 LEADER]: Becoming Leader. State: Replica: 3bd66da4382c45ffb3c5da2796583494, State: Running, Role: LEADER
I20260429 14:04:28.050305 8967 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bd66da4382c45ffb3c5da2796583494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd66da4382c45ffb3c5da2796583494" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 37963 } }
I20260429 14:04:28.050655 8968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bd66da4382c45ffb3c5da2796583494 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bd66da4382c45ffb3c5da2796583494" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd66da4382c45ffb3c5da2796583494" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 37963 } } }
I20260429 14:04:28.050740 8968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bd66da4382c45ffb3c5da2796583494 [sys.catalog]: This master's current role is: LEADER
I20260429 14:04:28.050836 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 8897
I20260429 14:04:28.050956 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:04:28.051146 2839 external_mini_cluster.cc:1468] Setting key fe83f96fe208cb2ce0a59c96d1a5c8b7
I20260429 14:04:28.051580 8969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bd66da4382c45ffb3c5da2796583494 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bd66da4382c45ffb3c5da2796583494. Latest consensus state: current_term: 1 leader_uuid: "3bd66da4382c45ffb3c5da2796583494" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd66da4382c45ffb3c5da2796583494" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 37963 } } }
I20260429 14:04:28.051659 8969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bd66da4382c45ffb3c5da2796583494 [sys.catalog]: This master's current role is: LEADER
W20260429 14:04:28.052258 8984 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 3bd66da4382c45ffb3c5da2796583494: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260429 14:04:28.052312 8984 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260429 14:04:28.052500 8974 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:04:28.052925 8974 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:04:28.055490 8974 catalog_manager.cc:1357] Generated new cluster ID: 2b4fdda126df44da948d320d89a34e14
I20260429 14:04:28.055542 8974 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:04:28.056731 8985 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:28.051810 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:38588 (local address 127.2.197.254:37963)
0429 14:04:28.051997 (+ 187us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:28.052005 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:28.052056 (+ 51us) server_negotiation.cc:408] Connection header received
0429 14:04:28.052232 (+ 176us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:28.052239 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:28.052299 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:28.052449 (+ 150us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:28.053049 (+ 600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:28.054431 (+ 1382us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:28.055238 (+ 807us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:28.055496 (+ 258us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:28.056107 (+ 611us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:28.056114 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:28.056114 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:04:28.056346 (+ 232us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:28.056370 (+ 24us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:04:28.056379 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:28.056457 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:28.056526 (+ 69us) server_negotiation.cc:300] Negotiation successful
0429 14:04:28.056582 (+ 56us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
I20260429 14:04:28.057844 2839 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260429 14:04:28.071024 8974 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:04:28.072034 8974 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:04:28.079131 8974 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 3bd66da4382c45ffb3c5da2796583494: Generated new TSK 0
I20260429 14:04:28.079358 8974 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 14:04:28.132819 8985 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:28.086408 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:38602 (local address 127.2.197.254:37963)
0429 14:04:28.086572 (+ 164us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:28.086574 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:28.086591 (+ 17us) server_negotiation.cc:408] Connection header received
0429 14:04:28.086740 (+ 149us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:28.086742 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:28.086759 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:28.086835 (+ 76us) server_negotiation.cc:227] Negotiated authn=JWT
0429 14:04:28.087304 (+ 469us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:28.087921 (+ 617us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:28.088799 (+ 878us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:28.088969 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:28.089046 (+ 77us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0429 14:04:28.132579 (+ 43533us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0429 14:04:28.132742 (+ 163us) negotiation.cc:326] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38602: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":118}
W20260429 14:04:28.132910 8985 negotiation.cc:343] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38602: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
W20260429 14:04:28.132974 8993 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:28.086258 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37963 (local address 127.0.0.1:38602)
0429 14:04:28.086509 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:28.086524 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:28.086607 (+ 83us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:28.086869 (+ 262us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:28.086870 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:28.086872 (+ 2us) client_negotiation.cc:190] Negotiated authn=JWT
0429 14:04:28.087139 (+ 267us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:28.087141 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:28.088022 (+ 881us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:28.088023 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:28.088683 (+ 660us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:28.088685 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:28.088777 (+ 92us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:28.088783 (+ 6us) client_negotiation.cc:253] Sending JWT_EXCHANGE NegotiatePB request
0429 14:04:28.132706 (+ 43923us) client_negotiation.cc:284] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0429 14:04:28.132863 (+ 157us) negotiation.cc:326] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.197.254:37963: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":153,"threads_started":1}
I20260429 14:04:28.133399 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 8897
2026-04-29T14:04:28Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (221 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (86 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (84 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:04:28 dist-test-slave-8n77 krb5kdc[9005](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:04:28 dist-test-slave-8n77 krb5kdc[9005](info): set up 2 sockets
Apr 29 14:04:28 dist-test-slave-8n77 krb5kdc[9005](info): commencing operation
krb5kdc: starting...
W20260429 14:04:30.335875 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:04:30 dist-test-slave-8n77 krb5kdc[9005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:04:30Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:04:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:04:30.486632 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:40139
--webserver_interface=127.2.197.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:36605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:40139
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:04:30.564147 9021 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:30.564384 9021 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:30.564422 9021 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:30.566031 9021 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:04:30.566141 9021 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:30.566165 9021 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:30.566185 9021 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:04:30.566215 9021 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:04:30.568094 9021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:36605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:40139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:40139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:30.568410 9021 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:30.568619 9021 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:30.571388 9026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:30.571390 9029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:30.571444 9027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:30.571616 9021 server_base.cc:1061] running on GCE node
I20260429 14:04:30.571879 9021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:30.572129 9021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:30.573288 9021 hybrid_clock.cc:648] HybridClock initialized: now 1777471470573266 us; error 46 us; skew 500 ppm
Apr 29 14:04:30 dist-test-slave-8n77 krb5kdc[9005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471470, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:30.575970 9021 init.cc:377] Logged in from keytab as kudu/127.2.197.254@KRBTEST.COM (short username kudu)
I20260429 14:04:30.576558 9021 webserver.cc:492] Webserver started at http://127.2.197.254:38447/ using document root <none> and password file <none>
I20260429 14:04:30.576798 9021 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:30.576865 9021 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:30.576977 9021 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:04:30.578025 9021 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "9bcad98eb8a24be9be636b17f97059ac"
format_stamp: "Formatted at 2026-04-29 14:04:30 on dist-test-slave-8n77"
server_key: "9a76c809b206724e2705cb4dc86751fc"
server_key_iv: "217d32d70b9985d967e0ef32bd021361"
server_key_version: "encryptionkey@0"
I20260429 14:04:30.578435 9021 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "9bcad98eb8a24be9be636b17f97059ac"
format_stamp: "Formatted at 2026-04-29 14:04:30 on dist-test-slave-8n77"
server_key: "9a76c809b206724e2705cb4dc86751fc"
server_key_iv: "217d32d70b9985d967e0ef32bd021361"
server_key_version: "encryptionkey@0"
I20260429 14:04:30.579708 9021 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:04:30.580469 9036 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:30.580677 9021 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:04:30.580772 9021 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "9bcad98eb8a24be9be636b17f97059ac"
format_stamp: "Formatted at 2026-04-29 14:04:30 on dist-test-slave-8n77"
server_key: "9a76c809b206724e2705cb4dc86751fc"
server_key_iv: "217d32d70b9985d967e0ef32bd021361"
server_key_version: "encryptionkey@0"
I20260429 14:04:30.580839 9021 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:30.592100 9021 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:30.594398 9021 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:30.594529 9021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:30.598693 9021 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:40139
I20260429 14:04:30.598752 9088 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:40139 every 8 connection(s)
I20260429 14:04:30.599099 9021 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:04:30.599743 9089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:30.601164 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 9021
I20260429 14:04:30.601302 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:04:30.601475 2839 external_mini_cluster.cc:1468] Setting key b05ce223982c58640d2fe167e24d7bd6
I20260429 14:04:30.602643 9089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bcad98eb8a24be9be636b17f97059ac: Bootstrap starting.
I20260429 14:04:30.603324 9089 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bcad98eb8a24be9be636b17f97059ac: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:30.603642 9089 log.cc:826] T 00000000000000000000000000000000 P 9bcad98eb8a24be9be636b17f97059ac: Log is configured to *not* fsync() on all Append() calls
I20260429 14:04:30.604396 9089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bcad98eb8a24be9be636b17f97059ac: No bootstrap required, opened a new log
I20260429 14:04:30.606014 9089 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9bcad98eb8a24be9be636b17f97059ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcad98eb8a24be9be636b17f97059ac" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40139 } }
I20260429 14:04:30.606156 9089 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9bcad98eb8a24be9be636b17f97059ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:30.606191 9089 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9bcad98eb8a24be9be636b17f97059ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bcad98eb8a24be9be636b17f97059ac, State: Initialized, Role: FOLLOWER
I20260429 14:04:30.606350 9089 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bcad98eb8a24be9be636b17f97059ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcad98eb8a24be9be636b17f97059ac" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40139 } }
I20260429 14:04:30.606451 9089 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9bcad98eb8a24be9be636b17f97059ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:04:30.606494 9089 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9bcad98eb8a24be9be636b17f97059ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:04:30.606757 9089 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9bcad98eb8a24be9be636b17f97059ac [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:30.607404 9089 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9bcad98eb8a24be9be636b17f97059ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcad98eb8a24be9be636b17f97059ac" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40139 } }
I20260429 14:04:30.607549 9089 leader_election.cc:304] T 00000000000000000000000000000000 P 9bcad98eb8a24be9be636b17f97059ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bcad98eb8a24be9be636b17f97059ac; no voters:
I20260429 14:04:30.607741 9089 leader_election.cc:290] T 00000000000000000000000000000000 P 9bcad98eb8a24be9be636b17f97059ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:04:30.607784 9094 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9bcad98eb8a24be9be636b17f97059ac [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:04:30.607893 9094 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9bcad98eb8a24be9be636b17f97059ac [term 1 LEADER]: Becoming Leader. State: Replica: 9bcad98eb8a24be9be636b17f97059ac, State: Running, Role: LEADER
I20260429 14:04:30.607991 9089 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9bcad98eb8a24be9be636b17f97059ac [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:04:30.608028 9094 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bcad98eb8a24be9be636b17f97059ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcad98eb8a24be9be636b17f97059ac" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40139 } }
I20260429 14:04:30.608448 9096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bcad98eb8a24be9be636b17f97059ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bcad98eb8a24be9be636b17f97059ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcad98eb8a24be9be636b17f97059ac" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40139 } } }
Apr 29 14:04:30 dist-test-slave-8n77 krb5kdc[9005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:04:30.608552 9096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bcad98eb8a24be9be636b17f97059ac [sys.catalog]: This master's current role is: LEADER
I20260429 14:04:30.608434 9094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bcad98eb8a24be9be636b17f97059ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bcad98eb8a24be9be636b17f97059ac. Latest consensus state: current_term: 1 leader_uuid: "9bcad98eb8a24be9be636b17f97059ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcad98eb8a24be9be636b17f97059ac" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40139 } } }
I20260429 14:04:30.608875 9094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bcad98eb8a24be9be636b17f97059ac [sys.catalog]: This master's current role is: LEADER
I20260429 14:04:30.609297 9100 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:04:30.610251 9100 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:04:30.612062 9100 catalog_manager.cc:1357] Generated new cluster ID: 41ef61db5b5644fa877c6c00d99f4587
I20260429 14:04:30.612113 9100 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:04:30.613641 9092 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:30.602111 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33676 (local address 127.2.197.254:40139)
0429 14:04:30.602446 (+ 335us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:30.602462 (+ 16us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:30.602514 (+ 52us) server_negotiation.cc:408] Connection header received
0429 14:04:30.602896 (+ 382us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:30.602903 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:30.602945 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:30.603057 (+ 112us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:30.604033 (+ 976us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:30.604867 (+ 834us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:30.605736 (+ 869us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:30.606054 (+ 318us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:30.609208 (+ 3154us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:30.609368 (+ 160us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:30.609369 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:30.609401 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:30.612038 (+ 2637us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:30.612479 (+ 441us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:30.612483 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:30.612484 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:30.612547 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:30.612743 (+ 196us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:30.612744 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:30.612744 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:30.613058 (+ 314us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:30.613260 (+ 202us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:30.613359 (+ 99us) server_negotiation.cc:300] Negotiation successful
0429 14:04:30.613499 (+ 140us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
I20260429 14:04:30.641196 9100 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:04:30.641824 9100 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:04:30.649916 9100 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 9bcad98eb8a24be9be636b17f97059ac: Generated new TSK 0
I20260429 14:04:30.650166 9100 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:04:30.710836 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.193:0
--local_ip_for_outbound_sockets=127.2.197.193
--webserver_interface=127.2.197.193
--webserver_port=0
--tserver_master_addrs=127.2.197.254:40139
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:36605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:04:30.795838 9117 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:30.796072 9117 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:30.796113 9117 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:30.797662 9117 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:30.797761 9117 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:30.797813 9117 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.193
I20260429 14:04:30.799584 9117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:36605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.197.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:40139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.193
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:30.799885 9117 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:30.800086 9117 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:30.803205 9122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:30.803208 9123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:30.803226 9125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:30.803349 9117 server_base.cc:1061] running on GCE node
I20260429 14:04:30.803681 9117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:30.803941 9117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:30.805121 9117 hybrid_clock.cc:648] HybridClock initialized: now 1777471470805099 us; error 34 us; skew 500 ppm
Apr 29 14:04:30 dist-test-slave-8n77 krb5kdc[9005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471470, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:30.807760 9117 init.cc:377] Logged in from keytab as kudu/127.2.197.193@KRBTEST.COM (short username kudu)
I20260429 14:04:30.808331 9117 webserver.cc:492] Webserver started at http://127.2.197.193:34775/ using document root <none> and password file <none>
I20260429 14:04:30.808560 9117 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:30.808604 9117 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:30.808677 9117 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:04:30.809614 9117 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-0/data/instance:
uuid: "14385343f3c442f185b31da554851973"
format_stamp: "Formatted at 2026-04-29 14:04:30 on dist-test-slave-8n77"
server_key: "b9f6685eebf1ee9d7eda36483624b27c"
server_key_iv: "4774109ff1a71b9d1cd94bbd14b407ed"
server_key_version: "encryptionkey@0"
I20260429 14:04:30.810184 9117 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance:
uuid: "14385343f3c442f185b31da554851973"
format_stamp: "Formatted at 2026-04-29 14:04:30 on dist-test-slave-8n77"
server_key: "b9f6685eebf1ee9d7eda36483624b27c"
server_key_iv: "4774109ff1a71b9d1cd94bbd14b407ed"
server_key_version: "encryptionkey@0"
I20260429 14:04:30.811606 9117 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:04:30.812458 9132 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:30.812712 9117 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:04:30.812867 9117 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-0/wal
uuid: "14385343f3c442f185b31da554851973"
format_stamp: "Formatted at 2026-04-29 14:04:30 on dist-test-slave-8n77"
server_key: "b9f6685eebf1ee9d7eda36483624b27c"
server_key_iv: "4774109ff1a71b9d1cd94bbd14b407ed"
server_key_version: "encryptionkey@0"
I20260429 14:04:30.812943 9117 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:30.840019 9117 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:30.842100 9117 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:30.842219 9117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:30.842434 9117 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:04:30.842824 9117 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:04:30.842974 9117 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:30.843053 9117 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:04:30.843073 9117 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:30.849416 9117 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.193:41129
I20260429 14:04:30.849450 9245 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.193:41129 every 8 connection(s)
I20260429 14:04:30.849786 9117 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
Apr 29 14:04:30 dist-test-slave-8n77 krb5kdc[9005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471470, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:04:30.856678 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 9117
I20260429 14:04:30.856814 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance
I20260429 14:04:30.856987 2839 external_mini_cluster.cc:1468] Setting key 93dc4274c1dbc4b754f01c621c0e9856
I20260429 14:04:30.859846 9248 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:30.850595 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40139 (local address 127.2.197.193:52661)
0429 14:04:30.850762 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:30.850811 (+ 49us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:30.851272 (+ 461us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:30.851513 (+ 241us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:30.851517 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:30.851867 (+ 350us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:30.852142 (+ 275us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:30.852145 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:30.852910 (+ 765us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:30.852911 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:30.853384 (+ 473us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:30.853387 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:30.853490 (+ 103us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:30.854454 (+ 964us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:30.854463 (+ 9us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:30.856144 (+ 1681us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:30.858211 (+ 2067us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:30.858219 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:30.858222 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:30.858600 (+ 378us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:30.858813 (+ 213us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:30.858814 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:30.858814 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:30.858918 (+ 104us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:30.859289 (+ 371us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:30.859295 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:30.859454 (+ 159us) client_negotiation.cc:770] Sending connection context
0429 14:04:30.859545 (+ 91us) client_negotiation.cc:241] Negotiation successful
0429 14:04:30.859703 (+ 158us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
I20260429 14:04:30.859841 9092 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:30.850607 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:52661 (local address 127.2.197.254:40139)
0429 14:04:30.850683 (+ 76us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:30.850686 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:30.850931 (+ 245us) server_negotiation.cc:408] Connection header received
0429 14:04:30.851365 (+ 434us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:30.851367 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:30.851377 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:30.851436 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:30.852270 (+ 834us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:30.852807 (+ 537us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:30.853531 (+ 724us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:30.853676 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:30.856250 (+ 2574us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:30.856256 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:30.856256 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:30.856263 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:30.858118 (+ 1855us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:30.858708 (+ 590us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:30.858709 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:30.858709 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:30.858734 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:30.859029 (+ 295us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:30.859030 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:30.859030 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:30.859196 (+ 166us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:30.859293 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:30.859597 (+ 304us) server_negotiation.cc:300] Negotiation successful
0429 14:04:30.859723 (+ 126us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260429 14:04:30.860401 9246 heartbeater.cc:344] Connected to a master server at 127.2.197.254:40139
I20260429 14:04:30.860507 9246 heartbeater.cc:461] Registering TS with master...
I20260429 14:04:30.860713 9246 heartbeater.cc:507] Master 127.2.197.254:40139 requested a full tablet report, sending...
I20260429 14:04:30.861249 9053 ts_manager.cc:194] Registered new tserver with Master: 14385343f3c442f185b31da554851973 (127.2.197.193:41129)
I20260429 14:04:30.862303 9053 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.193@KRBTEST.COM'} at 127.2.197.193:52661
WARNING: no policy specified for kudu/127.2.197.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:04:30.915292 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.194:0
--local_ip_for_outbound_sockets=127.2.197.194
--webserver_interface=127.2.197.194
--webserver_port=0
--tserver_master_addrs=127.2.197.254:40139
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:36605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:04:30.996709 9253 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:30.996968 9253 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:30.997011 9253 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:30.998598 9253 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:30.998688 9253 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:30.998733 9253 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.194
I20260429 14:04:31.000406 9253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:36605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.197.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:40139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.194
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:31.000706 9253 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:31.000909 9253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:31.003675 9261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:31.003808 9259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:31.003891 9258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:31.004004 9253 server_base.cc:1061] running on GCE node
I20260429 14:04:31.004189 9253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:31.004400 9253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:31.005661 9253 hybrid_clock.cc:648] HybridClock initialized: now 1777471471005661 us; error 95 us; skew 500 ppm
Apr 29 14:04:31 dist-test-slave-8n77 krb5kdc[9005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471471, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:31.008400 9253 init.cc:377] Logged in from keytab as kudu/127.2.197.194@KRBTEST.COM (short username kudu)
I20260429 14:04:31.009017 9253 webserver.cc:492] Webserver started at http://127.2.197.194:33779/ using document root <none> and password file <none>
I20260429 14:04:31.009305 9253 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:31.009349 9253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:31.009450 9253 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:04:31.010684 9253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-1/data/instance:
uuid: "aa7f9880aa9d4fdf9b45d3a92337d361"
format_stamp: "Formatted at 2026-04-29 14:04:31 on dist-test-slave-8n77"
server_key: "a5b504b9d1206f02a1745cb00ec0f491"
server_key_iv: "16c00bc0704c98debed12dae20282cf1"
server_key_version: "encryptionkey@0"
I20260429 14:04:31.011260 9253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance:
uuid: "aa7f9880aa9d4fdf9b45d3a92337d361"
format_stamp: "Formatted at 2026-04-29 14:04:31 on dist-test-slave-8n77"
server_key: "a5b504b9d1206f02a1745cb00ec0f491"
server_key_iv: "16c00bc0704c98debed12dae20282cf1"
server_key_version: "encryptionkey@0"
I20260429 14:04:31.012923 9253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260429 14:04:31.014262 9268 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:31.014576 9253 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:31.014662 9253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-1/wal
uuid: "aa7f9880aa9d4fdf9b45d3a92337d361"
format_stamp: "Formatted at 2026-04-29 14:04:31 on dist-test-slave-8n77"
server_key: "a5b504b9d1206f02a1745cb00ec0f491"
server_key_iv: "16c00bc0704c98debed12dae20282cf1"
server_key_version: "encryptionkey@0"
I20260429 14:04:31.014744 9253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:31.027112 9253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:31.029209 9253 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:31.029321 9253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:31.029515 9253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:04:31.029842 9253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:04:31.029877 9253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:31.029899 9253 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:04:31.029912 9253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:31.035787 9253 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.194:46673
I20260429 14:04:31.035857 9381 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.194:46673 every 8 connection(s)
I20260429 14:04:31.036144 9253 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
I20260429 14:04:31.040576 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 9253
I20260429 14:04:31.040716 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance
I20260429 14:04:31.041069 2839 external_mini_cluster.cc:1468] Setting key 8f9f2e93fb0a45288b5e769a24eadebb
Apr 29 14:04:31 dist-test-slave-8n77 krb5kdc[9005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471471, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:04:31.045863 9092 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:31.037050 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:48447 (local address 127.2.197.254:40139)
0429 14:04:31.037138 (+ 88us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:31.037140 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:31.037214 (+ 74us) server_negotiation.cc:408] Connection header received
0429 14:04:31.037834 (+ 620us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:31.037836 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:31.037847 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:31.037908 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:31.038847 (+ 939us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:31.039330 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:31.039946 (+ 616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:31.040072 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:31.042901 (+ 2829us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:31.042905 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:31.042906 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:31.042914 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:31.044431 (+ 1517us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:31.044918 (+ 487us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:31.044919 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:31.044920 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:31.044942 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:31.045174 (+ 232us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:31.045175 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:31.045175 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:31.045298 (+ 123us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:31.045353 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:31.045709 (+ 356us) server_negotiation.cc:300] Negotiation successful
0429 14:04:31.045796 (+ 87us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260429 14:04:31.046062 9384 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:31.036948 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40139 (local address 127.2.197.194:48447)
0429 14:04:31.037099 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:31.037133 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:31.037686 (+ 553us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:31.038000 (+ 314us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:31.038004 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:31.038440 (+ 436us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:31.038718 (+ 278us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:31.038721 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:31.039434 (+ 713us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:31.039435 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:31.039849 (+ 414us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:31.039850 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:31.039947 (+ 97us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:31.041210 (+ 1263us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:31.041222 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:31.042806 (+ 1584us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:31.044519 (+ 1713us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:31.044525 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:31.044528 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:31.044829 (+ 301us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:31.045011 (+ 182us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:31.045012 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:31.045013 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:31.045099 (+ 86us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:31.045366 (+ 267us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:31.045371 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:31.045540 (+ 169us) client_negotiation.cc:770] Sending connection context
0429 14:04:31.045668 (+ 128us) client_negotiation.cc:241] Negotiation successful
0429 14:04:31.045826 (+ 158us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20260429 14:04:31.046726 9382 heartbeater.cc:344] Connected to a master server at 127.2.197.254:40139
I20260429 14:04:31.046849 9382 heartbeater.cc:461] Registering TS with master...
I20260429 14:04:31.047065 9382 heartbeater.cc:507] Master 127.2.197.254:40139 requested a full tablet report, sending...
I20260429 14:04:31.047502 9053 ts_manager.cc:194] Registered new tserver with Master: aa7f9880aa9d4fdf9b45d3a92337d361 (127.2.197.194:46673)
I20260429 14:04:31.047996 9053 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.194@KRBTEST.COM'} at 127.2.197.194:48447
WARNING: no policy specified for kudu/127.2.197.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:04:31.093433 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.195:0
--local_ip_for_outbound_sockets=127.2.197.195
--webserver_interface=127.2.197.195
--webserver_port=0
--tserver_master_addrs=127.2.197.254:40139
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:36605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:04:31.192626 9389 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:31.192809 9389 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:31.192834 9389 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:31.194772 9389 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:31.194832 9389 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:31.194875 9389 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.195
I20260429 14:04:31.196592 9389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:36605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.197.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:40139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.195
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:31.196862 9389 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:31.197050 9389 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:31.199765 9395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:31.199810 9394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:31.199791 9397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:31.200011 9389 server_base.cc:1061] running on GCE node
I20260429 14:04:31.200188 9389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:31.200441 9389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:31.201562 9389 hybrid_clock.cc:648] HybridClock initialized: now 1777471471201549 us; error 32 us; skew 500 ppm
Apr 29 14:04:31 dist-test-slave-8n77 krb5kdc[9005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471471, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:31.204136 9389 init.cc:377] Logged in from keytab as kudu/127.2.197.195@KRBTEST.COM (short username kudu)
I20260429 14:04:31.204731 9389 webserver.cc:492] Webserver started at http://127.2.197.195:40935/ using document root <none> and password file <none>
I20260429 14:04:31.204939 9389 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:31.205003 9389 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:31.205108 9389 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:04:31.206037 9389 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-2/data/instance:
uuid: "845bc4f9928746ecac55074288afe554"
format_stamp: "Formatted at 2026-04-29 14:04:31 on dist-test-slave-8n77"
server_key: "565c0fd1fe5cab0a38cb35e8c586e959"
server_key_iv: "b9299b62ac7dd78d84a5582f794835f2"
server_key_version: "encryptionkey@0"
I20260429 14:04:31.206429 9389 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance:
uuid: "845bc4f9928746ecac55074288afe554"
format_stamp: "Formatted at 2026-04-29 14:04:31 on dist-test-slave-8n77"
server_key: "565c0fd1fe5cab0a38cb35e8c586e959"
server_key_iv: "b9299b62ac7dd78d84a5582f794835f2"
server_key_version: "encryptionkey@0"
I20260429 14:04:31.211478 9389 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.000s sys 0.002s
I20260429 14:04:31.212244 9404 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:31.212401 9389 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260429 14:04:31.212469 9389 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-2/wal
uuid: "845bc4f9928746ecac55074288afe554"
format_stamp: "Formatted at 2026-04-29 14:04:31 on dist-test-slave-8n77"
server_key: "565c0fd1fe5cab0a38cb35e8c586e959"
server_key_iv: "b9299b62ac7dd78d84a5582f794835f2"
server_key_version: "encryptionkey@0"
I20260429 14:04:31.212525 9389 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:31.238610 9389 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:31.240720 9389 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:31.240890 9389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:31.241150 9389 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:04:31.241484 9389 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:04:31.241541 9389 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:31.241592 9389 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:04:31.241629 9389 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:31.249460 9389 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.195:39509
I20260429 14:04:31.249485 9517 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.195:39509 every 8 connection(s)
I20260429 14:04:31.249832 9389 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
Apr 29 14:04:31 dist-test-slave-8n77 krb5kdc[9005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471471, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:04:31.258770 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 9389
I20260429 14:04:31.258876 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance
I20260429 14:04:31.259063 2839 external_mini_cluster.cc:1468] Setting key 7c7625fbd476812012e11fc2efacc373
I20260429 14:04:31.260329 9092 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:31.250649 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:53573 (local address 127.2.197.254:40139)
0429 14:04:31.250726 (+ 77us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:31.250728 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:31.250966 (+ 238us) server_negotiation.cc:408] Connection header received
0429 14:04:31.251718 (+ 752us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:31.251720 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:31.251730 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:31.251782 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:31.252918 (+ 1136us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:31.253401 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:31.254367 (+ 966us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:31.254493 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:31.257387 (+ 2894us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:31.257392 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:31.257392 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:31.257400 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:31.258794 (+ 1394us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:31.259310 (+ 516us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:31.259311 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:31.259312 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:31.259343 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:31.259625 (+ 282us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:31.259626 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:31.259626 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:31.259774 (+ 148us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:31.259822 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:31.260182 (+ 360us) server_negotiation.cc:300] Negotiation successful
0429 14:04:31.260260 (+ 78us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260429 14:04:31.260582 9520 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:31.250613 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40139 (local address 127.2.197.195:53573)
0429 14:04:31.250827 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:31.250870 (+ 43us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:31.251602 (+ 732us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:31.251857 (+ 255us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:31.251862 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:31.252350 (+ 488us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:31.252786 (+ 436us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:31.252791 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:31.253509 (+ 718us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:31.253510 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:31.254226 (+ 716us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:31.254228 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:31.254411 (+ 183us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:31.255505 (+ 1094us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:31.255519 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:31.257280 (+ 1761us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:31.258932 (+ 1652us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:31.258940 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:31.258943 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:31.259224 (+ 281us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:31.259440 (+ 216us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:31.259441 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:31.259441 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:31.259545 (+ 104us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:31.259871 (+ 326us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:31.259876 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:31.260034 (+ 158us) client_negotiation.cc:770] Sending connection context
0429 14:04:31.260159 (+ 125us) client_negotiation.cc:241] Negotiation successful
0429 14:04:31.260336 (+ 177us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":91,"threads_started":1}
I20260429 14:04:31.261242 9518 heartbeater.cc:344] Connected to a master server at 127.2.197.254:40139
I20260429 14:04:31.261340 9518 heartbeater.cc:461] Registering TS with master...
I20260429 14:04:31.261579 9518 heartbeater.cc:507] Master 127.2.197.254:40139 requested a full tablet report, sending...
I20260429 14:04:31.262007 9053 ts_manager.cc:194] Registered new tserver with Master: 845bc4f9928746ecac55074288afe554 (127.2.197.195:39509)
I20260429 14:04:31.262485 9053 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.195@KRBTEST.COM'} at 127.2.197.195:53573
I20260429 14:04:31.270859 2839 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/test-admin.keytab.
Apr 29 14:04:31 dist-test-slave-8n77 krb5kdc[9005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:31.287434 2839 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260429 14:04:31.289250 9525 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:31.288395 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:45642 (local address 127.2.197.193:41129)
0429 14:04:31.288599 (+ 204us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:31.288608 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:31.288621 (+ 13us) server_negotiation.cc:408] Connection header received
0429 14:04:31.288655 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:31.288658 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:31.288677 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:31.288784 (+ 107us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:31.289152 (+ 368us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45642: BlockingRecv error: recv got EOF from 127.0.0.1:45642 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":110,"threads_started":1}
W20260429 14:04:31.289369 2839 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/krb5cc)
Apr 29 14:04:31 dist-test-slave-8n77 krb5kdc[9005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:31.290652 2839 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Apr 29 14:04:31 dist-test-slave-8n77 krb5kdc[9005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:31.291824 2839 init.cc:295] Successfully reacquired a new kerberos TGT
Apr 29 14:04:31 dist-test-slave-8n77 krb5kdc[9005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.193@KRBTEST.COM
I20260429 14:04:31.301180 9525 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:31.292517 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:45658 (local address 127.2.197.193:41129)
0429 14:04:31.292584 (+ 67us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:31.292586 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:31.292648 (+ 62us) server_negotiation.cc:408] Connection header received
0429 14:04:31.292701 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:31.292703 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:31.292717 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:31.292795 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:31.293372 (+ 577us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:31.294742 (+ 1370us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:31.295337 (+ 595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:31.295615 (+ 278us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:31.297800 (+ 2185us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:31.297840 (+ 40us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:31.297840 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:31.297849 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:31.300089 (+ 2240us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:31.300411 (+ 322us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:31.300414 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:31.300415 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:31.300447 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:31.300572 (+ 125us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:31.300573 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:31.300573 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:31.300700 (+ 127us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:31.300858 (+ 158us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:31.300907 (+ 49us) server_negotiation.cc:300] Negotiation successful
0429 14:04:31.301015 (+ 108us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260429 14:04:31.303030 9525 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:31.302489 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:45662 (local address 127.2.197.193:41129)
0429 14:04:31.302546 (+ 57us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:31.302548 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:31.302576 (+ 28us) server_negotiation.cc:408] Connection header received
0429 14:04:31.302635 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:31.302636 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:31.302649 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:31.302706 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:31.302971 (+ 265us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45662: BlockingRecv error: recv got EOF from 127.0.0.1:45662 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260429 14:04:31.303115 2839 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/krb5cc' of type FILE
Apr 29 14:04:31 dist-test-slave-8n77 krb5kdc[9005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:31.304276 2839 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Apr 29 14:04:31 dist-test-slave-8n77 krb5kdc[9005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:31.305365 2839 init.cc:295] Successfully reacquired a new kerberos TGT
Apr 29 14:04:31 dist-test-slave-8n77 krb5kdc[9005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.193@KRBTEST.COM
I20260429 14:04:31.313446 9525 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:31.305992 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:45664 (local address 127.2.197.193:41129)
0429 14:04:31.306050 (+ 58us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:31.306052 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:31.306106 (+ 54us) server_negotiation.cc:408] Connection header received
0429 14:04:31.306151 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:31.306152 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:31.306163 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:31.306230 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:31.306807 (+ 577us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:31.307684 (+ 877us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:31.308222 (+ 538us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:31.308447 (+ 225us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:31.310695 (+ 2248us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:31.310699 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:31.310700 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:31.310707 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:31.312334 (+ 1627us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:31.312662 (+ 328us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:31.312663 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:31.312664 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:31.312689 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:31.312906 (+ 217us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:31.312907 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:31.312908 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:31.313013 (+ 105us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:31.313110 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:31.313217 (+ 107us) server_negotiation.cc:300] Negotiation successful
0429 14:04:31.313315 (+ 98us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260429 14:04:31.315374 9525 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:31.314697 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:45676 (local address 127.2.197.193:41129)
0429 14:04:31.314736 (+ 39us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:31.314738 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:31.314882 (+ 144us) server_negotiation.cc:408] Connection header received
0429 14:04:31.314914 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:31.314915 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:31.314927 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:31.315006 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:31.315310 (+ 304us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45676: BlockingRecv error: recv got EOF from 127.0.0.1:45676 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260429 14:04:31.315480 2839 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777471413505459-2839-0/krb5kdc/krb5cc' of type FILE
Apr 29 14:04:31 dist-test-slave-8n77 krb5kdc[9005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:31.316834 2839 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Apr 29 14:04:31 dist-test-slave-8n77 krb5kdc[9005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:31.317953 2839 init.cc:295] Successfully reacquired a new kerberos TGT
Apr 29 14:04:31 dist-test-slave-8n77 krb5kdc[9005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.193@KRBTEST.COM
I20260429 14:04:31.325259 9525 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:31.318667 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:45684 (local address 127.2.197.193:41129)
0429 14:04:31.318733 (+ 66us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:31.318736 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:31.318789 (+ 53us) server_negotiation.cc:408] Connection header received
0429 14:04:31.318838 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:31.318839 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:31.318851 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:31.318933 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:31.319394 (+ 461us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:31.320179 (+ 785us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:31.320805 (+ 626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:31.320970 (+ 165us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:31.322954 (+ 1984us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:31.322957 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:31.322958 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:31.322965 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:31.324344 (+ 1379us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:31.324676 (+ 332us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:31.324677 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:31.324677 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:31.324699 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:31.324870 (+ 171us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:31.324871 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:31.324871 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:31.324950 (+ 79us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:31.325023 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:31.325128 (+ 105us) server_negotiation.cc:300] Negotiation successful
0429 14:04:31.325195 (+ 67us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260429 14:04:31.325947 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 9117
I20260429 14:04:31.332129 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 9253
I20260429 14:04:31.337628 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 9389
I20260429 14:04:31.342175 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 9021
2026-04-29T14:04:31Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3040 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:04:31 dist-test-slave-8n77 krb5kdc[9538](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:04:31 dist-test-slave-8n77 krb5kdc[9538](info): set up 2 sockets
Apr 29 14:04:31 dist-test-slave-8n77 krb5kdc[9538](info): commencing operation
krb5kdc: starting...
W20260429 14:04:33.392552 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.030s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:04:33 dist-test-slave-8n77 krb5kdc[9538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:04:33Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:04:33Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab.
I20260429 14:04:33.538420 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:40335
--webserver_interface=127.2.197.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.197.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:34323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:40335
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:04:33.614440 9554 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:33.614651 9554 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:33.614675 9554 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:33.616282 9554 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:04:33.616343 9554 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:33.616360 9554 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:33.616374 9554 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:04:33.616434 9554 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:04:33.618160 9554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:34323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:40335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.197.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:40335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:33.618482 9554 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:33.618695 9554 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:33.621783 9560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:33.621950 9554 server_base.cc:1061] running on GCE node
W20260429 14:04:33.621989 9562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:33.621771 9559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:33.622404 9554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:33.622737 9554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:33.623914 9554 hybrid_clock.cc:648] HybridClock initialized: now 1777471473623890 us; error 41 us; skew 500 ppm
Apr 29 14:04:33 dist-test-slave-8n77 krb5kdc[9538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471473, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:33.626479 9554 init.cc:377] Logged in from keytab as oryx/127.2.197.254@KRBTEST.COM (short username oryx)
I20260429 14:04:33.627038 9554 webserver.cc:492] Webserver started at http://127.2.197.254:36015/ using document root <none> and password file <none>
I20260429 14:04:33.627240 9554 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:33.627301 9554 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:33.627419 9554 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:04:33.628352 9554 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "f8a2e74b05bf42de93b558952af4f587"
format_stamp: "Formatted at 2026-04-29 14:04:33 on dist-test-slave-8n77"
server_key: "bf229d5e6a4df809271055cc1063ef22"
server_key_iv: "b614b85895f44b85f9eec2fd802d933b"
server_key_version: "encryptionkey@0"
I20260429 14:04:33.628685 9554 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "f8a2e74b05bf42de93b558952af4f587"
format_stamp: "Formatted at 2026-04-29 14:04:33 on dist-test-slave-8n77"
server_key: "bf229d5e6a4df809271055cc1063ef22"
server_key_iv: "b614b85895f44b85f9eec2fd802d933b"
server_key_version: "encryptionkey@0"
I20260429 14:04:33.629936 9554 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260429 14:04:33.630759 9569 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:33.630947 9554 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260429 14:04:33.631024 9554 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "f8a2e74b05bf42de93b558952af4f587"
format_stamp: "Formatted at 2026-04-29 14:04:33 on dist-test-slave-8n77"
server_key: "bf229d5e6a4df809271055cc1063ef22"
server_key_iv: "b614b85895f44b85f9eec2fd802d933b"
server_key_version: "encryptionkey@0"
I20260429 14:04:33.631104 9554 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:33.643026 9554 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:33.650518 9554 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:33.650693 9554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:33.654740 9554 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:40335
I20260429 14:04:33.654790 9621 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:40335 every 8 connection(s)
I20260429 14:04:33.655107 9554 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:04:33.655699 9622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:33.658010 9622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587: Bootstrap starting.
I20260429 14:04:33.658579 9622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:33.658795 9622 log.cc:826] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587: Log is configured to *not* fsync() on all Append() calls
I20260429 14:04:33.659762 9622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587: No bootstrap required, opened a new log
I20260429 14:04:33.660984 9622 raft_consensus.cc:359] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } }
I20260429 14:04:33.661114 9622 raft_consensus.cc:385] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:33.661135 9622 raft_consensus.cc:740] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8a2e74b05bf42de93b558952af4f587, State: Initialized, Role: FOLLOWER
I20260429 14:04:33.661293 9622 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } }
I20260429 14:04:33.661384 9622 raft_consensus.cc:399] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:04:33.661409 9622 raft_consensus.cc:493] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:04:33.661435 9622 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:33.661994 9622 raft_consensus.cc:515] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } }
I20260429 14:04:33.662083 9622 leader_election.cc:304] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8a2e74b05bf42de93b558952af4f587; no voters:
I20260429 14:04:33.662212 9622 leader_election.cc:290] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:04:33.662340 9625 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:04:33.662515 9625 raft_consensus.cc:697] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [term 1 LEADER]: Becoming Leader. State: Replica: f8a2e74b05bf42de93b558952af4f587, State: Running, Role: LEADER
I20260429 14:04:33.662523 9622 sys_catalog.cc:565] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:04:33.662623 9625 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } }
I20260429 14:04:33.663015 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 9554
I20260429 14:04:33.663126 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:04:33.663304 2839 external_mini_cluster.cc:1468] Setting key 9508b7744067d2230d3a7fe63a49c508
I20260429 14:04:33.663834 9626 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8a2e74b05bf42de93b558952af4f587" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } }
I20260429 14:04:33.663929 9626 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [sys.catalog]: This master's current role is: LEADER
I20260429 14:04:33.664162 9635 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:04:33.664219 9627 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8a2e74b05bf42de93b558952af4f587. Latest consensus state: current_term: 1 leader_uuid: "f8a2e74b05bf42de93b558952af4f587" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } }
I20260429 14:04:33.664400 9627 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [sys.catalog]: This master's current role is: LEADER
I20260429 14:04:33.664927 9635 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:04:33.667420 9635 catalog_manager.cc:1357] Generated new cluster ID: bfb901df49664bdeb1e28c49a5039180
I20260429 14:04:33.667501 9635 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Apr 29 14:04:33 dist-test-slave-8n77 krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM
I20260429 14:04:33.675441 9636 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:33.664040 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:59184 (local address 127.2.197.254:40335)
0429 14:04:33.664187 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:33.664287 (+ 100us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:33.664720 (+ 433us) server_negotiation.cc:408] Connection header received
0429 14:04:33.664845 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:33.664852 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:33.664902 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:33.666574 (+ 1672us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:33.667020 (+ 446us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:33.668191 (+ 1171us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:33.668997 (+ 806us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:33.669231 (+ 234us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:33.671805 (+ 2574us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:33.671828 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:33.671829 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:33.671842 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:33.673866 (+ 2024us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:33.674310 (+ 444us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:33.674312 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:33.674313 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:33.674373 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:33.674565 (+ 192us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:33.674566 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:33.674566 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:33.674836 (+ 270us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:33.674958 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:33.675134 (+ 176us) server_negotiation.cc:300] Negotiation successful
0429 14:04:33.675260 (+ 126us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":63,"threads_started":1}
I20260429 14:04:33.676970 9635 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:04:33.677639 9635 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:04:33.697613 9635 catalog_manager.cc:6044] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587: Generated new TSK 0
I20260429 14:04:33.697844 9635 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab.
I20260429 14:04:33.762629 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.193:0
--local_ip_for_outbound_sockets=127.2.197.193
--webserver_interface=127.2.197.193
--webserver_port=0
--tserver_master_addrs=127.2.197.254:40335
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.197.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:34323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:04:33.849284 9650 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:33.849495 9650 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:33.849522 9650 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:33.851258 9650 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:33.851327 9650 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:33.851357 9650 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.193
I20260429 14:04:33.853228 9650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:34323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.197.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.197.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:40335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.193
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:33.853456 9650 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:33.853607 9650 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:33.856396 9656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:33.856401 9658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:33.856400 9655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:33.856746 9650 server_base.cc:1061] running on GCE node
I20260429 14:04:33.857060 9650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:33.857321 9650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:33.858525 9650 hybrid_clock.cc:648] HybridClock initialized: now 1777471473858497 us; error 95 us; skew 500 ppm
Apr 29 14:04:33 dist-test-slave-8n77 krb5kdc[9538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471473, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:33.861230 9650 init.cc:377] Logged in from keytab as oryx/127.2.197.193@KRBTEST.COM (short username oryx)
I20260429 14:04:33.861900 9650 webserver.cc:492] Webserver started at http://127.2.197.193:43183/ using document root <none> and password file <none>
I20260429 14:04:33.862107 9650 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:33.862169 9650 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:33.862323 9650 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:04:33.863363 9650 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-0/data/instance:
uuid: "468ecc83d41743dc981739ecb9b20067"
format_stamp: "Formatted at 2026-04-29 14:04:33 on dist-test-slave-8n77"
server_key: "114e5cdd04bc80c925a8fe57de0b8972"
server_key_iv: "b57daf51b0cfce9932059690d6e0ce44"
server_key_version: "encryptionkey@0"
I20260429 14:04:33.863766 9650 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance:
uuid: "468ecc83d41743dc981739ecb9b20067"
format_stamp: "Formatted at 2026-04-29 14:04:33 on dist-test-slave-8n77"
server_key: "114e5cdd04bc80c925a8fe57de0b8972"
server_key_iv: "b57daf51b0cfce9932059690d6e0ce44"
server_key_version: "encryptionkey@0"
I20260429 14:04:33.865255 9650 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260429 14:04:33.866135 9665 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:33.866394 9650 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:33.866495 9650 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-0/wal
uuid: "468ecc83d41743dc981739ecb9b20067"
format_stamp: "Formatted at 2026-04-29 14:04:33 on dist-test-slave-8n77"
server_key: "114e5cdd04bc80c925a8fe57de0b8972"
server_key_iv: "b57daf51b0cfce9932059690d6e0ce44"
server_key_version: "encryptionkey@0"
I20260429 14:04:33.866827 9650 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:33.903034 9650 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:33.905324 9650 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:33.905571 9650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:33.905844 9650 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:04:33.906177 9650 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:04:33.906234 9650 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:33.906315 9650 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:04:33.906340 9650 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:33.914748 9650 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.193:45535
I20260429 14:04:33.914841 9778 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.193:45535 every 8 connection(s)
I20260429 14:04:33.915325 9650 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
I20260429 14:04:33.918767 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 9650
I20260429 14:04:33.918910 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance
I20260429 14:04:33.919067 2839 external_mini_cluster.cc:1468] Setting key 3b6476f72e96aae30f82d47df421a358
Apr 29 14:04:33 dist-test-slave-8n77 krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471473, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM
I20260429 14:04:33.925330 9781 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:33.916367 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.2.197.193:34181)
0429 14:04:33.916511 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:33.916546 (+ 35us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:33.917048 (+ 502us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:33.917312 (+ 264us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:33.917316 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:33.917620 (+ 304us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:33.917915 (+ 295us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:33.917918 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:33.918760 (+ 842us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:33.918761 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:33.919349 (+ 588us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:33.919351 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:33.919544 (+ 193us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:33.920507 (+ 963us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:33.920515 (+ 8us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:33.921958 (+ 1443us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:33.923980 (+ 2022us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:33.923987 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:33.923990 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:33.924215 (+ 225us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:33.924403 (+ 188us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:33.924403 (+ 0us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:33.924403 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:33.924459 (+ 56us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:33.924800 (+ 341us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:33.924805 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:33.924939 (+ 134us) client_negotiation.cc:770] Sending connection context
0429 14:04:33.925037 (+ 98us) client_negotiation.cc:241] Negotiation successful
0429 14:04:33.925153 (+ 116us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":78,"threads_started":1}
I20260429 14:04:33.925328 9636 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:33.916370 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:34181 (local address 127.2.197.254:40335)
0429 14:04:33.916446 (+ 76us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:33.916448 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:33.916715 (+ 267us) server_negotiation.cc:408] Connection header received
0429 14:04:33.917157 (+ 442us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:33.917158 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:33.917169 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:33.917231 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:33.918015 (+ 784us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:33.918641 (+ 626us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:33.919756 (+ 1115us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:33.919945 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:33.922058 (+ 2113us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:33.922062 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:33.922063 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:33.922070 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:33.923878 (+ 1808us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:33.924299 (+ 421us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:33.924300 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:33.924301 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:33.924332 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:33.924566 (+ 234us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:33.924567 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:33.924568 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:33.924709 (+ 141us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:33.924790 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:33.925084 (+ 294us) server_negotiation.cc:300] Negotiation successful
0429 14:04:33.925190 (+ 106us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260429 14:04:33.925935 9779 heartbeater.cc:344] Connected to a master server at 127.2.197.254:40335
I20260429 14:04:33.926045 9779 heartbeater.cc:461] Registering TS with master...
I20260429 14:04:33.926708 9779 heartbeater.cc:507] Master 127.2.197.254:40335 requested a full tablet report, sending...
I20260429 14:04:33.927503 9585 ts_manager.cc:194] Registered new tserver with Master: 468ecc83d41743dc981739ecb9b20067 (127.2.197.193:45535)
WARNING: no policy specified for oryx/127.2.197.194@KRBTEST.COM; defaulting to no policy
I20260429 14:04:33.928320 9585 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.197.193@KRBTEST.COM'} at 127.2.197.193:34181
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.2.197.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab.
I20260429 14:04:33.979596 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.194:0
--local_ip_for_outbound_sockets=127.2.197.194
--webserver_interface=127.2.197.194
--webserver_port=0
--tserver_master_addrs=127.2.197.254:40335
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.197.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:34323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:04:34.061378 9786 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:34.061647 9786 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:34.061688 9786 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:34.063513 9786 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:34.063583 9786 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:34.063630 9786 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.194
I20260429 14:04:34.065510 9786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:34323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.197.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.197.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:40335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.194
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:34.065780 9786 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:34.065975 9786 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:34.068897 9792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:34.068955 9791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:34.068965 9794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:34.069389 9786 server_base.cc:1061] running on GCE node
I20260429 14:04:34.069629 9786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:34.069828 9786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:34.070987 9786 hybrid_clock.cc:648] HybridClock initialized: now 1777471474070972 us; error 30 us; skew 500 ppm
Apr 29 14:04:34 dist-test-slave-8n77 krb5kdc[9538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471474, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:34.073839 9786 init.cc:377] Logged in from keytab as oryx/127.2.197.194@KRBTEST.COM (short username oryx)
I20260429 14:04:34.074491 9786 webserver.cc:492] Webserver started at http://127.2.197.194:41023/ using document root <none> and password file <none>
I20260429 14:04:34.074709 9786 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:34.074760 9786 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:34.074934 9786 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:04:34.075978 9786 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-1/data/instance:
uuid: "01baffeacfa6457094831efd2c408c4e"
format_stamp: "Formatted at 2026-04-29 14:04:34 on dist-test-slave-8n77"
server_key: "5b0f185c4316936dd9771540025742f9"
server_key_iv: "e8b9dfba9be31c2e4117899e0b35e2f6"
server_key_version: "encryptionkey@0"
I20260429 14:04:34.076308 9786 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance:
uuid: "01baffeacfa6457094831efd2c408c4e"
format_stamp: "Formatted at 2026-04-29 14:04:34 on dist-test-slave-8n77"
server_key: "5b0f185c4316936dd9771540025742f9"
server_key_iv: "e8b9dfba9be31c2e4117899e0b35e2f6"
server_key_version: "encryptionkey@0"
I20260429 14:04:34.077558 9786 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260429 14:04:34.078395 9801 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:34.078603 9786 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:34.078687 9786 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-1/wal
uuid: "01baffeacfa6457094831efd2c408c4e"
format_stamp: "Formatted at 2026-04-29 14:04:34 on dist-test-slave-8n77"
server_key: "5b0f185c4316936dd9771540025742f9"
server_key_iv: "e8b9dfba9be31c2e4117899e0b35e2f6"
server_key_version: "encryptionkey@0"
I20260429 14:04:34.078769 9786 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:34.101950 9786 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:34.104113 9786 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:34.104291 9786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:34.104547 9786 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:04:34.104868 9786 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:04:34.104925 9786 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:34.104961 9786 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:04:34.105008 9786 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:34.111060 9786 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.194:44633
I20260429 14:04:34.111107 9914 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.194:44633 every 8 connection(s)
I20260429 14:04:34.111431 9786 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
I20260429 14:04:34.115504 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 9786
I20260429 14:04:34.115660 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance
I20260429 14:04:34.115864 2839 external_mini_cluster.cc:1468] Setting key 71253276693cb947f35d3f6a287d68d3
Apr 29 14:04:34 dist-test-slave-8n77 krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471474, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM
I20260429 14:04:34.122372 9636 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.112295 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:59459 (local address 127.2.197.254:40335)
0429 14:04:34.112364 (+ 69us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:34.112366 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:34.112576 (+ 210us) server_negotiation.cc:408] Connection header received
0429 14:04:34.113055 (+ 479us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:34.113056 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:34.113067 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:34.113132 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:34.113906 (+ 774us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.114436 (+ 530us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:34.115058 (+ 622us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.115188 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:34.118796 (+ 3608us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:34.118801 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:34.118801 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:34.118811 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:34.120786 (+ 1975us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:34.121375 (+ 589us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:34.121376 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:34.121377 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:34.121410 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:34.121672 (+ 262us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:34.121673 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:34.121673 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:34.121779 (+ 106us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:34.121872 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:34.122157 (+ 285us) server_negotiation.cc:300] Negotiation successful
0429 14:04:34.122255 (+ 98us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260429 14:04:34.122534 9917 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.112272 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.2.197.194:59459)
0429 14:04:34.112450 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:34.112490 (+ 40us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:34.112967 (+ 477us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:34.113194 (+ 227us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:34.113199 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:34.113539 (+ 340us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:34.113799 (+ 260us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:34.113803 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.114563 (+ 760us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:34.114564 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:34.114961 (+ 397us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:34.114974 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.115080 (+ 106us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:34.116342 (+ 1262us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:34.116357 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:34.118684 (+ 2327us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:34.120916 (+ 2232us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:34.120922 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:34.120925 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:34.121210 (+ 285us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:34.121480 (+ 270us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:34.121481 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:34.121481 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:34.121596 (+ 115us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:34.121865 (+ 269us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:34.121870 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:34.121998 (+ 128us) client_negotiation.cc:770] Sending connection context
0429 14:04:34.122135 (+ 137us) client_negotiation.cc:241] Negotiation successful
0429 14:04:34.122314 (+ 179us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":101,"threads_started":1}
I20260429 14:04:34.123344 9915 heartbeater.cc:344] Connected to a master server at 127.2.197.254:40335
I20260429 14:04:34.123445 9915 heartbeater.cc:461] Registering TS with master...
I20260429 14:04:34.123687 9915 heartbeater.cc:507] Master 127.2.197.254:40335 requested a full tablet report, sending...
I20260429 14:04:34.124418 9585 ts_manager.cc:194] Registered new tserver with Master: 01baffeacfa6457094831efd2c408c4e (127.2.197.194:44633)
I20260429 14:04:34.124929 9585 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.197.194@KRBTEST.COM'} at 127.2.197.194:59459
WARNING: no policy specified for oryx/127.2.197.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.2.197.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab.
I20260429 14:04:34.178378 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.195:0
--local_ip_for_outbound_sockets=127.2.197.195
--webserver_interface=127.2.197.195
--webserver_port=0
--tserver_master_addrs=127.2.197.254:40335
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.197.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:34323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:04:34.256325 9922 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:34.256507 9922 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:34.256532 9922 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:34.258016 9922 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:34.258056 9922 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:34.258082 9922 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.195
I20260429 14:04:34.259689 9922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:34323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.197.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.197.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:40335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.195
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:34.259874 9922 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:34.260008 9922 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:34.262766 9927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:34.262771 9928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:34.262805 9922 server_base.cc:1061] running on GCE node
W20260429 14:04:34.262801 9930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:34.263221 9922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:34.263451 9922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:34.264616 9922 hybrid_clock.cc:648] HybridClock initialized: now 1777471474264614 us; error 38 us; skew 500 ppm
Apr 29 14:04:34 dist-test-slave-8n77 krb5kdc[9538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471474, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:34.267215 9922 init.cc:377] Logged in from keytab as oryx/127.2.197.195@KRBTEST.COM (short username oryx)
I20260429 14:04:34.267783 9922 webserver.cc:492] Webserver started at http://127.2.197.195:46127/ using document root <none> and password file <none>
I20260429 14:04:34.267988 9922 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:34.268033 9922 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:34.268147 9922 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:04:34.269115 9922 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-2/data/instance:
uuid: "3e025d72db7c49159529617974c08232"
format_stamp: "Formatted at 2026-04-29 14:04:34 on dist-test-slave-8n77"
server_key: "40e0bf989a9a5e56f54349278d9f4326"
server_key_iv: "eb04b8fc89fd8b5f0ab564f9e58f6568"
server_key_version: "encryptionkey@0"
I20260429 14:04:34.269462 9922 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance:
uuid: "3e025d72db7c49159529617974c08232"
format_stamp: "Formatted at 2026-04-29 14:04:34 on dist-test-slave-8n77"
server_key: "40e0bf989a9a5e56f54349278d9f4326"
server_key_iv: "eb04b8fc89fd8b5f0ab564f9e58f6568"
server_key_version: "encryptionkey@0"
I20260429 14:04:34.270823 9922 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260429 14:04:34.271595 9937 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:34.271803 9922 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:04:34.271862 9922 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-2/wal
uuid: "3e025d72db7c49159529617974c08232"
format_stamp: "Formatted at 2026-04-29 14:04:34 on dist-test-slave-8n77"
server_key: "40e0bf989a9a5e56f54349278d9f4326"
server_key_iv: "eb04b8fc89fd8b5f0ab564f9e58f6568"
server_key_version: "encryptionkey@0"
I20260429 14:04:34.271947 9922 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:34.292480 9922 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:34.294780 9922 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:34.294925 9922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:34.295179 9922 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:04:34.295506 9922 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:04:34.295595 9922 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:34.295641 9922 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:04:34.295674 9922 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:34.301440 9922 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.195:42039
I20260429 14:04:34.301533 10050 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.195:42039 every 8 connection(s)
I20260429 14:04:34.301843 9922 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
I20260429 14:04:34.303293 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 9922
I20260429 14:04:34.303377 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance
I20260429 14:04:34.303563 2839 external_mini_cluster.cc:1468] Setting key 6aca95b2b0b0747cdf69630da7b5690c
Apr 29 14:04:34 dist-test-slave-8n77 krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471474, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.195@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM
I20260429 14:04:34.311656 9636 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.302713 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:33065 (local address 127.2.197.254:40335)
0429 14:04:34.302780 (+ 67us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:34.302782 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:34.302950 (+ 168us) server_negotiation.cc:408] Connection header received
0429 14:04:34.303474 (+ 524us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:34.303476 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:34.303490 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:34.303644 (+ 154us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:34.304626 (+ 982us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.305102 (+ 476us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:34.305674 (+ 572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.305782 (+ 108us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:34.308877 (+ 3095us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:34.308881 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:34.308882 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:34.308889 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:34.310464 (+ 1575us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:34.310862 (+ 398us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:34.310863 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:34.310863 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:34.310883 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:34.311044 (+ 161us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:34.311044 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:34.311045 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:34.311134 (+ 89us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:34.311182 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:34.311497 (+ 315us) server_negotiation.cc:300] Negotiation successful
0429 14:04:34.311587 (+ 90us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260429 14:04:34.311762 10053 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.302672 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.2.197.195:33065)
0429 14:04:34.302858 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:34.302882 (+ 24us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:34.303318 (+ 436us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:34.303652 (+ 334us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:34.303656 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:34.304130 (+ 474us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:34.304514 (+ 384us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:34.304519 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.305197 (+ 678us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:34.305199 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:34.305598 (+ 399us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:34.305599 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.305695 (+ 96us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:34.306659 (+ 964us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:34.306668 (+ 9us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:34.308751 (+ 2083us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:34.310555 (+ 1804us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:34.310560 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:34.310562 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:34.310793 (+ 231us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:34.310939 (+ 146us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:34.310939 (+ 0us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:34.310940 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:34.310997 (+ 57us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:34.311230 (+ 233us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:34.311234 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:34.311347 (+ 113us) client_negotiation.cc:770] Sending connection context
0429 14:04:34.311438 (+ 91us) client_negotiation.cc:241] Negotiation successful
0429 14:04:34.311559 (+ 121us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":68,"threads_started":1}
I20260429 14:04:34.312393 10051 heartbeater.cc:344] Connected to a master server at 127.2.197.254:40335
I20260429 14:04:34.312567 10051 heartbeater.cc:461] Registering TS with master...
I20260429 14:04:34.312774 10051 heartbeater.cc:507] Master 127.2.197.254:40335 requested a full tablet report, sending...
I20260429 14:04:34.313206 9585 ts_manager.cc:194] Registered new tserver with Master: 3e025d72db7c49159529617974c08232 (127.2.197.195:42039)
I20260429 14:04:34.313572 9585 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.197.195@KRBTEST.COM'} at 127.2.197.195:33065
I20260429 14:04:34.315508 2839 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.197.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.197.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.197.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.2.197.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.2.197.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.2.197.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab.
I20260429 14:04:34.369876 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.253:44465
--webserver_interface=127.2.197.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.197.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:34323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:40335,127.2.197.253:44465
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:04:34.448633 10058 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:34.448853 10058 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:34.448879 10058 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:34.450449 10058 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:04:34.450517 10058 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:34.450534 10058 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:34.450548 10058 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:04:34.450562 10058 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:04:34.452275 10058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:34323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:40335,127.2.197.253:44465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.197.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.253:44465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.197.253
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
W20260429 14:04:34.452389 10058 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.2.197.254:40335,127.2.197.253:44465'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260429 14:04:34.452596 10058 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:34.452802 10058 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260429 14:04:34.455711 10058 server_base.cc:1061] running on GCE node
W20260429 14:04:34.455818 10066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:34.455834 10063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:34.455974 10064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:34.456193 10058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:34.456456 10058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:34.457630 10058 hybrid_clock.cc:648] HybridClock initialized: now 1777471474457605 us; error 36 us; skew 500 ppm
Apr 29 14:04:34 dist-test-slave-8n77 krb5kdc[9538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471474, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:34.460311 10058 init.cc:377] Logged in from keytab as oryx/127.2.197.253@KRBTEST.COM (short username oryx)
I20260429 14:04:34.460911 10058 webserver.cc:492] Webserver started at http://127.2.197.253:44235/ using document root <none> and password file <none>
I20260429 14:04:34.461103 10058 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:34.461159 10058 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:34.461261 10058 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:04:34.462206 10058 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-1/data/instance:
uuid: "7de05a456e054b9f83790eb862246127"
format_stamp: "Formatted at 2026-04-29 14:04:34 on dist-test-slave-8n77"
server_key: "4835e174a2638eaace43438d8c4a5dc5"
server_key_iv: "ef2f8369cb4bdf7fc530d6fb311d200a"
server_key_version: "encryptionkey@0"
I20260429 14:04:34.462592 10058 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-1/wal/instance:
uuid: "7de05a456e054b9f83790eb862246127"
format_stamp: "Formatted at 2026-04-29 14:04:34 on dist-test-slave-8n77"
server_key: "4835e174a2638eaace43438d8c4a5dc5"
server_key_iv: "ef2f8369cb4bdf7fc530d6fb311d200a"
server_key_version: "encryptionkey@0"
I20260429 14:04:34.463858 10058 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:34.464884 10073 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:34.465126 10058 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:34.465188 10058 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-1/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-1/wal
uuid: "7de05a456e054b9f83790eb862246127"
format_stamp: "Formatted at 2026-04-29 14:04:34 on dist-test-slave-8n77"
server_key: "4835e174a2638eaace43438d8c4a5dc5"
server_key_iv: "ef2f8369cb4bdf7fc530d6fb311d200a"
server_key_version: "encryptionkey@0"
I20260429 14:04:34.465232 10058 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:34.490211 10058 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:34.492236 10058 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:34.492360 10058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:34.496274 10058 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.253:44465
I20260429 14:04:34.496515 10125 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.253:44465 every 8 connection(s)
I20260429 14:04:34.496841 10058 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-1/data/info.pb
I20260429 14:04:34.497426 10126 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:34.499225 10126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } has no permanent_uuid. Determining permanent_uuid...
Apr 29 14:04:34 dist-test-slave-8n77 krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471474, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.253@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM
I20260429 14:04:34.504948 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 10058
I20260429 14:04:34.508038 9636 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.500001 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:59198 (local address 127.2.197.254:40335)
0429 14:04:34.500086 (+ 85us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:34.500088 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:34.500255 (+ 167us) server_negotiation.cc:408] Connection header received
0429 14:04:34.500646 (+ 391us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:34.500647 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:34.500660 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:34.500708 (+ 48us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:34.501430 (+ 722us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.501918 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:34.502523 (+ 605us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.502693 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:34.504643 (+ 1950us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:34.504647 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:34.504647 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:34.504655 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:34.506471 (+ 1816us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:34.506911 (+ 440us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:34.506912 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:34.506913 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:34.506948 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:34.507268 (+ 320us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:34.507269 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:34.507270 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:34.507445 (+ 175us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:34.507539 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:34.507824 (+ 285us) server_negotiation.cc:300] Negotiation successful
0429 14:04:34.507931 (+ 107us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260429 14:04:34.508128 10128 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.499970 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.0.0.1:59198)
0429 14:04:34.500150 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:34.500177 (+ 27us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:34.500550 (+ 373us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:34.500785 (+ 235us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:34.500788 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:34.501072 (+ 284us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:34.501325 (+ 253us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:34.501328 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.502010 (+ 682us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:34.502011 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:34.502442 (+ 431us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:34.502444 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.502586 (+ 142us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:34.503154 (+ 568us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:34.503163 (+ 9us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:34.504553 (+ 1390us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:34.506552 (+ 1999us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:34.506557 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:34.506560 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:34.506829 (+ 269us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:34.507047 (+ 218us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:34.507048 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:34.507048 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:34.507114 (+ 66us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:34.507561 (+ 447us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:34.507566 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:34.507680 (+ 114us) client_negotiation.cc:770] Sending connection context
0429 14:04:34.507781 (+ 101us) client_negotiation.cc:241] Negotiation successful
0429 14:04:34.507896 (+ 115us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":90,"threads_started":1}
I20260429 14:04:34.508744 10126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } has no permanent_uuid. Determining permanent_uuid...
Apr 29 14:04:34 dist-test-slave-8n77 krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.197.253@KRBTEST.COM
Apr 29 14:04:34 dist-test-slave-8n77 krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471474, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.253@KRBTEST.COM for oryx/127.2.197.253@KRBTEST.COM
I20260429 14:04:34.514854 10130 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.505472 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:46948 (local address 127.2.197.253:44465)
0429 14:04:34.505663 (+ 191us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:34.505670 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:34.505714 (+ 44us) server_negotiation.cc:408] Connection header received
0429 14:04:34.505809 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:34.505812 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:34.505824 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:34.505882 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:34.506765 (+ 883us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.507897 (+ 1132us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:34.508720 (+ 823us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.508905 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:34.511385 (+ 2480us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:34.511392 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:34.511393 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:34.511399 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:34.513560 (+ 2161us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:34.513945 (+ 385us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:34.513947 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:34.513948 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:34.513973 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:34.514194 (+ 221us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:34.514195 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:34.514195 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:34.514371 (+ 176us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:34.514491 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:34.514668 (+ 177us) server_negotiation.cc:300] Negotiation successful
0429 14:04:34.514743 (+ 75us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":127,"threads_started":1}
I20260429 14:04:34.516582 10128 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.509090 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.253:44465 (local address 127.0.0.1:46962)
0429 14:04:34.509253 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:34.509264 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:34.509314 (+ 50us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:34.509617 (+ 303us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:34.509619 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:34.509760 (+ 141us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:34.509882 (+ 122us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:34.509883 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.510570 (+ 687us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:34.510571 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:34.510958 (+ 387us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:34.510959 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.511047 (+ 88us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:34.511429 (+ 382us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:34.511433 (+ 4us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:34.513005 (+ 1572us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:34.515708 (+ 2703us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:34.515709 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:34.515709 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:34.515848 (+ 139us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:34.516014 (+ 166us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:34.516015 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:34.516015 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:34.516041 (+ 26us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:34.516251 (+ 210us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:34.516252 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:34.516284 (+ 32us) client_negotiation.cc:770] Sending connection context
0429 14:04:34.516367 (+ 83us) client_negotiation.cc:241] Negotiation successful
0429 14:04:34.516467 (+ 100us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260429 14:04:34.516650 10131 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.509122 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:46962 (local address 127.2.197.253:44465)
0429 14:04:34.509264 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:34.509266 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:34.509298 (+ 32us) server_negotiation.cc:408] Connection header received
0429 14:04:34.509367 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:34.509444 (+ 77us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:34.509466 (+ 22us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:34.509563 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:34.509962 (+ 399us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.510493 (+ 531us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:34.511045 (+ 552us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.511226 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:34.513069 (+ 1843us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:34.513091 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:34.513092 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:34.513098 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:34.515616 (+ 2518us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:34.515925 (+ 309us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:34.515926 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:34.515927 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:34.515947 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:34.516094 (+ 147us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:34.516095 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:34.516095 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:34.516178 (+ 83us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:34.516247 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:34.516386 (+ 139us) server_negotiation.cc:300] Negotiation successful
0429 14:04:34.516532 (+ 146us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":125,"thread_start_us":99,"threads_started":1}
I20260429 14:04:34.517707 10126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127: Bootstrap starting.
I20260429 14:04:34.518217 10126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:34.518524 10126 log.cc:826] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127: Log is configured to *not* fsync() on all Append() calls
I20260429 14:04:34.519151 10126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127: No bootstrap required, opened a new log
I20260429 14:04:34.520171 10126 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } }
I20260429 14:04:34.520272 10126 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:34.520291 10126 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7de05a456e054b9f83790eb862246127, State: Initialized, Role: FOLLOWER
I20260429 14:04:34.520390 10126 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } }
I20260429 14:04:34.520619 10126 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:04:34.520644 10133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } } }
I20260429 14:04:34.520767 10133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [sys.catalog]: This master's current role is: FOLLOWER
W20260429 14:04:34.522553 10144 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260429 14:04:34.522625 10144 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260429 14:04:34.524521 10058 master_runner.cc:428] Detected that this master 7de05a456e054b9f83790eb862246127 is joining an existing cluster
I20260429 14:04:34.524588 10058 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.197.253:44465
I20260429 14:04:34.531478 10151 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.525917 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.253:44465 (local address 127.0.0.1:46968)
0429 14:04:34.526043 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:34.526053 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:34.526123 (+ 70us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:34.526309 (+ 186us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:34.526310 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:34.526520 (+ 210us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:34.526725 (+ 205us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:34.526727 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.527583 (+ 856us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:34.527584 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:34.527990 (+ 406us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:34.527992 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.528071 (+ 79us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:34.528604 (+ 533us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:34.528609 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:34.528911 (+ 302us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:34.530520 (+ 1609us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:34.530521 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:34.530522 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:34.530665 (+ 143us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:34.530844 (+ 179us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:34.530844 (+ 0us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:34.530845 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:34.530872 (+ 27us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:34.531090 (+ 218us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:34.531091 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:34.531138 (+ 47us) client_negotiation.cc:770] Sending connection context
0429 14:04:34.531222 (+ 84us) client_negotiation.cc:241] Negotiation successful
0429 14:04:34.531330 (+ 108us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20260429 14:04:34.531478 10131 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.525963 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:46968 (local address 127.2.197.253:44465)
0429 14:04:34.525997 (+ 34us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:34.525999 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:34.526124 (+ 125us) server_negotiation.cc:408] Connection header received
0429 14:04:34.526198 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:34.526199 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:34.526209 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:34.526282 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:34.526807 (+ 525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.527493 (+ 686us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:34.528065 (+ 572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.528253 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:34.528990 (+ 737us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:34.528993 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:34.528993 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:34.528999 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:34.530428 (+ 1429us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:34.530740 (+ 312us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:34.530741 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:34.530742 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:34.530764 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:34.530926 (+ 162us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:34.530927 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:34.530927 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:34.531028 (+ 101us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:34.531102 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:34.531241 (+ 139us) server_negotiation.cc:300] Negotiation successful
0429 14:04:34.531396 (+ 155us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"mutex_wait_us":82,"server-negotiator.queue_time_us":19}
I20260429 14:04:34.554888 9636 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.525771 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:59208 (local address 127.2.197.254:40335)
0429 14:04:34.525849 (+ 78us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:34.525852 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:34.525925 (+ 73us) server_negotiation.cc:408] Connection header received
0429 14:04:34.526042 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:34.526043 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:34.526057 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:34.526126 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:34.526915 (+ 789us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.527634 (+ 719us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:34.528444 (+ 810us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.528642 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:34.551945 (+ 23303us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:34.551952 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:34.551952 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:34.551963 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:34.553646 (+ 1683us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:34.554102 (+ 456us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:34.554103 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:34.554104 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:34.554129 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:34.554365 (+ 236us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:34.554366 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:34.554368 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:34.554492 (+ 124us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:34.554554 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:34.554734 (+ 180us) server_negotiation.cc:300] Negotiation successful
0429 14:04:34.554818 (+ 84us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260429 14:04:34.554939 10150 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.525675 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.0.0.1:59208)
0429 14:04:34.525849 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:34.525861 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:34.525952 (+ 91us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:34.526158 (+ 206us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:34.526159 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:34.526492 (+ 333us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:34.526783 (+ 291us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:34.526785 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.527732 (+ 947us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:34.527733 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:34.528350 (+ 617us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:34.528353 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.528451 (+ 98us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:34.551275 (+ 22824us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:34.551283 (+ 8us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:34.551726 (+ 443us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:34.553783 (+ 2057us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:34.553785 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:34.553785 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:34.553998 (+ 213us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:34.554208 (+ 210us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:34.554209 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:34.554209 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:34.554243 (+ 34us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:34.554559 (+ 316us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:34.554560 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:34.554623 (+ 63us) client_negotiation.cc:770] Sending connection context
0429 14:04:34.554691 (+ 68us) client_negotiation.cc:241] Negotiation successful
0429 14:04:34.554821 (+ 130us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":104,"threads_started":1}
Apr 29 14:04:34 dist-test-slave-8n77 krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471473, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.254@KRBTEST.COM for oryx/127.2.197.253@KRBTEST.COM
I20260429 14:04:34.564386 10131 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.557258 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:46970 (local address 127.2.197.253:44465)
0429 14:04:34.557301 (+ 43us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:34.557303 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:34.557397 (+ 94us) server_negotiation.cc:408] Connection header received
0429 14:04:34.557527 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:34.557528 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:34.557562 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:34.557626 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:34.558201 (+ 575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.558703 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:34.559217 (+ 514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.559407 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:34.561468 (+ 2061us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:34.561472 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:34.561472 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:34.561480 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:34.563100 (+ 1620us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:34.563526 (+ 426us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:34.563528 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:34.563528 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:34.563576 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:34.563774 (+ 198us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:34.563775 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:34.563776 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:34.563893 (+ 117us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:34.563979 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:34.564171 (+ 192us) server_negotiation.cc:300] Negotiation successful
0429 14:04:34.564260 (+ 89us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260429 14:04:34.564400 10152 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.557140 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.253:44465 (local address 127.0.0.1:46970)
0429 14:04:34.557282 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:34.557316 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:34.557422 (+ 106us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:34.557631 (+ 209us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:34.557635 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:34.557937 (+ 302us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:34.558109 (+ 172us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:34.558111 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.558777 (+ 666us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:34.558778 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:34.559135 (+ 357us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:34.559137 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.559223 (+ 86us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:34.559729 (+ 506us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:34.559733 (+ 4us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:34.561368 (+ 1635us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:34.563212 (+ 1844us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:34.563219 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:34.563221 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:34.563427 (+ 206us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:34.563670 (+ 243us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:34.563671 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:34.563671 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:34.563700 (+ 29us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:34.563968 (+ 268us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:34.563974 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:34.564046 (+ 72us) client_negotiation.cc:770] Sending connection context
0429 14:04:34.564147 (+ 101us) client_negotiation.cc:241] Negotiation successful
0429 14:04:34.564261 (+ 114us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
I20260429 14:04:34.565263 9585 catalog_manager.cc:7158] Initiating ChangeConfig request to add master 127.2.197.253:44465: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7de05a456e054b9f83790eb862246127"
member_type: NON_VOTER
last_known_addr {
host: "127.2.197.253"
port: 44465
}
attrs {
promote: true
}
}
dest_uuid: "f8a2e74b05bf42de93b558952af4f587"
cas_config_opid_index: -1
I20260429 14:04:34.565569 9585 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: NON_VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: true } }
I20260429 14:04:34.566099 10154 raft_consensus.cc:2955] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 7de05a456e054b9f83790eb862246127 (127.2.197.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: NON_VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: true } } }
I20260429 14:04:34.567018 10154 catalog_manager.cc:7162] Successfully completed master ChangeConfig request to add master 127.2.197.253:44465
I20260429 14:04:34.567109 10154 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f8a2e74b05bf42de93b558952af4f587" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: NON_VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: true } } }
I20260429 14:04:34.567175 10154 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [sys.catalog]: This master's current role is: LEADER
I20260429 14:04:34.567739 10058 master.cc:562] Master@127.2.197.253:44465 shutting down...
W20260429 14:04:34.568601 9572 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260429 14:04:34.570216 10058 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260429 14:04:34.570369 10058 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260429 14:04:34.570430 10058 tablet_replica.cc:333] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127: stopping tablet replica
I20260429 14:04:34.573045 10058 master.cc:584] Master@127.2.197.253:44465 shutdown complete.
I20260429 14:04:34.573110 10058 master_runner.cc:305] Clearing existing system tablet
I20260429 14:04:34.573248 10058 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:34.573305 10058 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:34.574105 10058 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260429 14:04:34.574682 10159 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:34.574880 10152 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.574752 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.253:44465 (local address 127.0.0.1:46980)
0429 14:04:34.574791 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:34.574841 (+ 50us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.253:44465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260429 14:04:34.574981 9572 consensus_peers.cc:597] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 -> Peer 7de05a456e054b9f83790eb862246127 (127.2.197.253:44465): Couldn't send request to peer 7de05a456e054b9f83790eb862246127. Status: Network error: Client connection negotiation failed: client connection to 127.2.197.253:44465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260429 14:04:34.575069 10058 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:34.575140 10058 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260429 14:04:34.575171 10058 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-1/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-1/wal
uuid: "7de05a456e054b9f83790eb862246127"
format_stamp: "Formatted at 2026-04-29 14:04:34 on dist-test-slave-8n77"
server_key: "4835e174a2638eaace43438d8c4a5dc5"
server_key_iv: "ef2f8369cb4bdf7fc530d6fb311d200a"
server_key_version: "encryptionkey@0"
WARNING: no policy specified for oryx/127.2.197.252@KRBTEST.COM; defaulting to no policy
I20260429 14:04:34.575826 10058 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:04:34.577919 10058 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260429 14:04:34.578027 10058 log.cc:1199] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260429 14:04:34.578619 10058 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127: Deleting consensus metadata
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.197.252@KRBTEST.COM" created.
I20260429 14:04:34.578812 10058 master_runner.cc:315] Copying system tablet from 127.2.197.254:40335
I20260429 14:04:34.579802 10058 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127: tablet copy: Beginning tablet copy session from remote peer at address 127.2.197.254:40335
I20260429 14:04:34.587901 9636 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.580226 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:59220 (local address 127.2.197.254:40335)
0429 14:04:34.580738 (+ 512us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:34.580741 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:34.580883 (+ 142us) server_negotiation.cc:408] Connection header received
0429 14:04:34.580980 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:34.580982 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:34.580994 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:34.581057 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:34.581506 (+ 449us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.582307 (+ 801us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:34.583209 (+ 902us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.583405 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:34.584609 (+ 1204us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:34.584613 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:34.584614 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:34.584621 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:34.586594 (+ 1973us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:34.587034 (+ 440us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:34.587036 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:34.587036 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:34.587063 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:34.587243 (+ 180us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:34.587244 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:34.587245 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:34.587394 (+ 149us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:34.587440 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:34.587678 (+ 238us) server_negotiation.cc:300] Negotiation successful
0429 14:04:34.587776 (+ 98us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487}
I20260429 14:04:34.588253 10165 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.580156 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.0.0.1:59220)
0429 14:04:34.580791 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:34.580801 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:34.580900 (+ 99us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:34.581059 (+ 159us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:34.581060 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:34.581245 (+ 185us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:34.581418 (+ 173us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:34.581420 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.582403 (+ 983us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:34.582404 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:34.583086 (+ 682us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:34.583090 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.583477 (+ 387us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:34.584036 (+ 559us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:34.584042 (+ 6us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:34.584465 (+ 423us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:34.586704 (+ 2239us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:34.586705 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:34.586706 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:34.586937 (+ 231us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:34.587134 (+ 197us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:34.587135 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:34.587136 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:34.587171 (+ 35us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:34.587497 (+ 326us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:34.587498 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:34.587549 (+ 51us) client_negotiation.cc:770] Sending connection context
0429 14:04:34.588003 (+ 454us) client_negotiation.cc:241] Negotiation successful
0429 14:04:34.588123 (+ 120us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":83,"threads_started":1}
I20260429 14:04:34.588624 9606 tablet_copy_service.cc:140] P f8a2e74b05bf42de93b558952af4f587: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7de05a456e054b9f83790eb862246127 ({username='oryx', principal='oryx/127.2.197.253@KRBTEST.COM'} at 127.0.0.1:59220)
I20260429 14:04:34.588711 9606 tablet_copy_service.cc:161] P f8a2e74b05bf42de93b558952af4f587: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7de05a456e054b9f83790eb862246127 at {username='oryx', principal='oryx/127.2.197.253@KRBTEST.COM'} at 127.0.0.1:59220: session id = 7de05a456e054b9f83790eb862246127-00000000000000000000000000000000
I20260429 14:04:34.589398 9606 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587: Tablet Copy: opened 0 blocks and 1 log segments
I20260429 14:04:34.589960 10058 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260429 14:04:34.591336 10058 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127: tablet copy: Starting download of 0 data blocks...
I20260429 14:04:34.591480 10058 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127: tablet copy: Starting download of 1 WAL segments...
I20260429 14:04:34.592636 10058 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260429 14:04:34.593689 9606 tablet_copy_service.cc:342] P f8a2e74b05bf42de93b558952af4f587: Request end of tablet copy session 7de05a456e054b9f83790eb862246127-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.197.253@KRBTEST.COM'} at 127.0.0.1:59220
I20260429 14:04:34.593851 9606 tablet_copy_service.cc:434] P f8a2e74b05bf42de93b558952af4f587: ending tablet copy session 7de05a456e054b9f83790eb862246127-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7de05a456e054b9f83790eb862246127
W20260429 14:04:34.596629 10058 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260429 14:04:34.597590 10058 file_cache.cc:492] Constructed file cache file cache with capacity 419430
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.197.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.2.197.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab.
W20260429 14:04:34.598891 10167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:34.599380 10170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:34.599490 10168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:34.600900 10058 server_base.cc:1061] running on GCE node
I20260429 14:04:34.601018 10058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:34.601163 10058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:34.602332 10058 hybrid_clock.cc:648] HybridClock initialized: now 1777471474602307 us; error 43 us; skew 500 ppm
Apr 29 14:04:34 dist-test-slave-8n77 krb5kdc[9538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471474, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:34.604033 10058 init.cc:377] Logged in from keytab as oryx/127.2.197.253@KRBTEST.COM (short username oryx)
I20260429 14:04:34.604328 10058 webserver.cc:492] Webserver started at http://127.2.197.253:43173/ using document root <none> and password file <none>
I20260429 14:04:34.604447 10058 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:34.604485 10058 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:34.605252 10058 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:34.605887 10178 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
WARNING: no policy specified for HTTP/127.2.197.252@KRBTEST.COM; defaulting to no policy
I20260429 14:04:34.606182 10058 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260429 14:04:34.606240 10058 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-1/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-1/wal
uuid: "7de05a456e054b9f83790eb862246127"
format_stamp: "Formatted at 2026-04-29 14:04:34 on dist-test-slave-8n77"
server_key: "4835e174a2638eaace43438d8c4a5dc5"
server_key_iv: "ef2f8369cb4bdf7fc530d6fb311d200a"
server_key_version: "encryptionkey@0"
I20260429 14:04:34.606340 10058 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.252@KRBTEST.COM" created.
I20260429 14:04:34.623176 10058 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.2.197.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab.
I20260429 14:04:34.630379 10058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:34.630965 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.252:36013
--webserver_interface=127.2.197.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.197.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:34323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:40335,127.2.197.253:44465,127.2.197.252:36013
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260429 14:04:34.634972 10058 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.253:44465
I20260429 14:04:34.635308 10232 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.253:44465 every 8 connection(s)
I20260429 14:04:34.635442 10058 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-1/data/info.pb
I20260429 14:04:34.636374 10233 sys_catalog.cc:263] Verifying existing consensus state
I20260429 14:04:34.637300 10233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127: Bootstrap starting.
I20260429 14:04:34.640038 10233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260429 14:04:34.640303 10233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127: Bootstrap complete.
I20260429 14:04:34.640789 10233 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: NON_VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: true } }
I20260429 14:04:34.640878 10233 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7de05a456e054b9f83790eb862246127, State: Initialized, Role: LEARNER
I20260429 14:04:34.640924 10233 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: NON_VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: true } }
I20260429 14:04:34.641088 10233 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:04:34.642809 10247 catalog_manager.cc:1269] Loaded cluster ID: bfb901df49664bdeb1e28c49a5039180
I20260429 14:04:34.642899 10247 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127: loading cluster ID for follower catalog manager: success
I20260429 14:04:34.644134 10247 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127: acquiring CA information for follower catalog manager: success
I20260429 14:04:34.644397 10247 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260429 14:04:34.647583 10236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: NON_VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: true } } }
I20260429 14:04:34.647768 10236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [sys.catalog]: This master's current role is: LEARNER
W20260429 14:04:34.744068 10197 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:34.744318 10197 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:34.744360 10197 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:34.746465 10197 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:04:34.746548 10197 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:34.746606 10197 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:34.746652 10197 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:04:34.746682 10197 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:04:34.748622 10197 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:34323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:40335,127.2.197.253:44465,127.2.197.252:36013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.197.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.252:36013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.197.252
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:34.748924 10197 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:34.749126 10197 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:34.752766 10255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:34.752883 10253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:34.753123 10197 server_base.cc:1061] running on GCE node
W20260429 14:04:34.752766 10252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:34.753441 10197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:34.753826 10197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:34.755021 10197 hybrid_clock.cc:648] HybridClock initialized: now 1777471474755005 us; error 49 us; skew 500 ppm
Apr 29 14:04:34 dist-test-slave-8n77 krb5kdc[9538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471474, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:34.758251 10197 init.cc:377] Logged in from keytab as oryx/127.2.197.252@KRBTEST.COM (short username oryx)
I20260429 14:04:34.759230 10197 webserver.cc:492] Webserver started at http://127.2.197.252:40853/ using document root <none> and password file <none>
I20260429 14:04:34.759459 10197 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:34.759528 10197 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:34.759666 10197 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:04:34.760624 10197 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-2/data/instance:
uuid: "9782bdb83d3143c6a2f310e1c7de398f"
format_stamp: "Formatted at 2026-04-29 14:04:34 on dist-test-slave-8n77"
server_key: "15e070571f9d9bb7e6242232ff6f415e"
server_key_iv: "3eeaeeb76e14b7201fc2019860c1b371"
server_key_version: "encryptionkey@0"
I20260429 14:04:34.760958 10197 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-2/wal/instance:
uuid: "9782bdb83d3143c6a2f310e1c7de398f"
format_stamp: "Formatted at 2026-04-29 14:04:34 on dist-test-slave-8n77"
server_key: "15e070571f9d9bb7e6242232ff6f415e"
server_key_iv: "3eeaeeb76e14b7201fc2019860c1b371"
server_key_version: "encryptionkey@0"
I20260429 14:04:34.762244 10197 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260429 14:04:34.763212 10262 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:34.763478 10197 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:34.763619 10197 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-2/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-2/wal
uuid: "9782bdb83d3143c6a2f310e1c7de398f"
format_stamp: "Formatted at 2026-04-29 14:04:34 on dist-test-slave-8n77"
server_key: "15e070571f9d9bb7e6242232ff6f415e"
server_key_iv: "3eeaeeb76e14b7201fc2019860c1b371"
server_key_version: "encryptionkey@0"
I20260429 14:04:34.763682 10197 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:34.777753 10197 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:34.779855 10197 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:34.780030 10197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:34.784303 10197 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.252:36013
I20260429 14:04:34.784379 10314 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.252:36013 every 8 connection(s)
I20260429 14:04:34.784663 10197 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-2/data/info.pb
I20260429 14:04:34.785249 10315 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:34.787056 10315 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } has no permanent_uuid. Determining permanent_uuid...
I20260429 14:04:34.790242 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 10197
Apr 29 14:04:34 dist-test-slave-8n77 krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471474, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.252@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM
I20260429 14:04:34.796435 9636 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.787886 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:59234 (local address 127.2.197.254:40335)
0429 14:04:34.787951 (+ 65us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:34.787954 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:34.788010 (+ 56us) server_negotiation.cc:408] Connection header received
0429 14:04:34.788416 (+ 406us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:34.788418 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:34.788429 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:34.788557 (+ 128us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:34.789279 (+ 722us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.789763 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:34.790508 (+ 745us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.790753 (+ 245us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:34.792937 (+ 2184us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:34.792941 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:34.792942 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:34.792948 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:34.794882 (+ 1934us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:34.795403 (+ 521us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:34.795405 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:34.795405 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:34.795434 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:34.795748 (+ 314us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:34.795749 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:34.795750 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:34.795844 (+ 94us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:34.795920 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:34.796282 (+ 362us) server_negotiation.cc:300] Negotiation successful
0429 14:04:34.796359 (+ 77us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260429 14:04:34.796629 10317 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.787704 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.0.0.1:59234)
0429 14:04:34.787874 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:34.787922 (+ 48us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:34.788319 (+ 397us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:34.788618 (+ 299us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:34.788622 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:34.788919 (+ 297us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:34.789169 (+ 250us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:34.789172 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.789853 (+ 681us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:34.789854 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:34.790408 (+ 554us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:34.790411 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.790538 (+ 127us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:34.791092 (+ 554us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:34.791101 (+ 9us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:34.792845 (+ 1744us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:34.794992 (+ 2147us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:34.794999 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:34.795001 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:34.795321 (+ 320us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:34.795540 (+ 219us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:34.795541 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:34.795541 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:34.795632 (+ 91us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:34.795934 (+ 302us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:34.795938 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:34.796108 (+ 170us) client_negotiation.cc:770] Sending connection context
0429 14:04:34.796236 (+ 128us) client_negotiation.cc:241] Negotiation successful
0429 14:04:34.796383 (+ 147us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":61,"threads_started":1}
Apr 29 14:04:34 dist-test-slave-8n77 krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.197.252@KRBTEST.COM
I20260429 14:04:34.797224 10315 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } has no permanent_uuid. Determining permanent_uuid...
I20260429 14:04:34.800712 10318 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.791152 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36550 (local address 127.2.197.252:36013)
0429 14:04:34.791311 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:34.791325 (+ 14us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:34.791338 (+ 13us) server_negotiation.cc:408] Connection header received
0429 14:04:34.791355 (+ 17us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:34.791357 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:34.791371 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:34.791445 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:34.792458 (+ 1013us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.793516 (+ 1058us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:34.794382 (+ 866us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.795047 (+ 665us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:34.797803 (+ 2756us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:34.797811 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:34.797812 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:34.797825 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:34.799528 (+ 1703us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:34.799935 (+ 407us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:34.799938 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:34.799938 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:34.799968 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:34.800157 (+ 189us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:34.800159 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:34.800159 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:34.800263 (+ 104us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:34.800406 (+ 143us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:34.800486 (+ 80us) server_negotiation.cc:300] Negotiation successful
0429 14:04:34.800585 (+ 99us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":109,"threads_started":1}
Apr 29 14:04:34 dist-test-slave-8n77 krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471474, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.252@KRBTEST.COM for oryx/127.2.197.253@KRBTEST.COM
I20260429 14:04:34.804540 10317 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.797553 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.253:44465 (local address 127.0.0.1:46992)
0429 14:04:34.797604 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:34.797615 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:34.797653 (+ 38us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:34.797907 (+ 254us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:34.797908 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:34.798047 (+ 139us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:34.798179 (+ 132us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:34.798180 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.799147 (+ 967us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:34.799148 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:34.799541 (+ 393us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:34.799543 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.799697 (+ 154us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:34.800129 (+ 432us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:34.800134 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:34.801461 (+ 1327us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:34.803221 (+ 1760us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:34.803222 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:34.803223 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:34.803494 (+ 271us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:34.803894 (+ 400us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:34.803896 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:34.803896 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:34.803936 (+ 40us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:34.804200 (+ 264us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:34.804201 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:34.804252 (+ 51us) client_negotiation.cc:770] Sending connection context
0429 14:04:34.804329 (+ 77us) client_negotiation.cc:241] Negotiation successful
0429 14:04:34.804442 (+ 113us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260429 14:04:34.804584 10319 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.797642 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:46992 (local address 127.2.197.253:44465)
0429 14:04:34.797790 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:34.797792 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:34.797798 (+ 6us) server_negotiation.cc:408] Connection header received
0429 14:04:34.797818 (+ 20us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:34.797818 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:34.797828 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:34.797904 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:34.798268 (+ 364us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.798965 (+ 697us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:34.799696 (+ 731us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.799914 (+ 218us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:34.801557 (+ 1643us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:34.801562 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:34.801562 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:34.801570 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:34.803131 (+ 1561us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:34.803606 (+ 475us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:34.803608 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:34.803608 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:34.803648 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:34.804012 (+ 364us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:34.804013 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:34.804013 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:34.804129 (+ 116us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:34.804212 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:34.804353 (+ 141us) server_negotiation.cc:300] Negotiation successful
0429 14:04:34.804447 (+ 94us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":76,"threads_started":1}
I20260429 14:04:34.805342 10315 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.197.252" port: 36013 } has no permanent_uuid. Determining permanent_uuid...
Apr 29 14:04:34 dist-test-slave-8n77 krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471474, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.252@KRBTEST.COM for oryx/127.2.197.252@KRBTEST.COM
I20260429 14:04:34.811973 10317 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.805913 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:36013 (local address 127.0.0.1:36562)
0429 14:04:34.805969 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:34.805987 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:34.806024 (+ 37us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:34.806135 (+ 111us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:34.806136 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:34.806251 (+ 115us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:34.806443 (+ 192us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:34.806445 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.807088 (+ 643us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:34.807089 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:34.807542 (+ 453us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:34.807544 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.807614 (+ 70us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:34.808125 (+ 511us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:34.808130 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:34.809297 (+ 1167us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:34.810841 (+ 1544us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:34.810842 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:34.810842 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:34.810977 (+ 135us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:34.811267 (+ 290us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:34.811268 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:34.811269 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:34.811332 (+ 63us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:34.811700 (+ 368us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:34.811700 (+ 0us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:34.811742 (+ 42us) client_negotiation.cc:770] Sending connection context
0429 14:04:34.811805 (+ 63us) client_negotiation.cc:241] Negotiation successful
0429 14:04:34.811893 (+ 88us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260429 14:04:34.812166 10318 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.805968 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36562 (local address 127.2.197.252:36013)
0429 14:04:34.806024 (+ 56us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:34.806026 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:34.806034 (+ 8us) server_negotiation.cc:408] Connection header received
0429 14:04:34.806066 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:34.806067 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:34.806077 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:34.806123 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:34.806533 (+ 410us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.806983 (+ 450us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:34.807652 (+ 669us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.807869 (+ 217us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:34.809374 (+ 1505us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:34.809377 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:34.809378 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:34.809384 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:34.810744 (+ 1360us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:34.811069 (+ 325us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:34.811070 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:34.811071 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:34.811091 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:34.811537 (+ 446us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:34.811538 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:34.811538 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:34.811628 (+ 90us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:34.811688 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:34.811959 (+ 271us) server_negotiation.cc:300] Negotiation successful
0429 14:04:34.812055 (+ 96us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260429 14:04:34.813514 10315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f: Bootstrap starting.
I20260429 14:04:34.814415 10315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:34.814682 10315 log.cc:826] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f: Log is configured to *not* fsync() on all Append() calls
I20260429 14:04:34.815433 10315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f: No bootstrap required, opened a new log
I20260429 14:04:34.816726 10315 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 36013 } }
I20260429 14:04:34.816849 10315 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:34.816874 10315 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9782bdb83d3143c6a2f310e1c7de398f, State: Initialized, Role: FOLLOWER
I20260429 14:04:34.816990 10315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 36013 } }
I20260429 14:04:34.817271 10321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 36013 } } }
I20260429 14:04:34.817312 10315 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:04:34.817373 10321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [sys.catalog]: This master's current role is: FOLLOWER
W20260429 14:04:34.819413 10332 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260429 14:04:34.819495 10332 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260429 14:04:34.821777 10197 master_runner.cc:428] Detected that this master 9782bdb83d3143c6a2f310e1c7de398f is joining an existing cluster
I20260429 14:04:34.821856 10197 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.197.252:36013
I20260429 14:04:34.856621 10152 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.853197 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.0.0.1:59242)
0429 14:04:34.853276 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:34.853286 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:34.853323 (+ 37us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:34.853972 (+ 649us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:34.853973 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:34.853974 (+ 1us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:04:34.854162 (+ 188us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:34.854164 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.855345 (+ 1181us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:34.855346 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:34.856310 (+ 964us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:34.856313 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.856367 (+ 54us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:34.856424 (+ 57us) client_negotiation.cc:770] Sending connection context
0429 14:04:34.856466 (+ 42us) client_negotiation.cc:241] Negotiation successful
0429 14:04:34.856523 (+ 57us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260429 14:04:34.857738 10341 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.853426 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:59242 (local address 127.2.197.254:40335)
0429 14:04:34.853795 (+ 369us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:34.853797 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:34.853810 (+ 13us) server_negotiation.cc:408] Connection header received
0429 14:04:34.853850 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:34.853851 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:34.853865 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:34.853970 (+ 105us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:04:34.854425 (+ 455us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.855255 (+ 830us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:34.857081 (+ 1826us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.857560 (+ 479us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:34.857588 (+ 28us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:34.857613 (+ 25us) server_negotiation.cc:300] Negotiation successful
0429 14:04:34.857651 (+ 38us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":128,"threads_started":1}
I20260429 14:04:34.859169 10318 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.823004 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36576 (local address 127.2.197.252:36013)
0429 14:04:34.823049 (+ 45us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:34.823051 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:34.823270 (+ 219us) server_negotiation.cc:408] Connection header received
0429 14:04:34.823343 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:34.823344 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:34.823352 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:34.823403 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:34.851283 (+ 27880us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.851962 (+ 679us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:34.852696 (+ 734us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.852866 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:34.855285 (+ 2419us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:34.855288 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:34.855289 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:34.855295 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:34.857182 (+ 1887us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:34.857746 (+ 564us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:34.857747 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:34.857747 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:34.857771 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:34.858588 (+ 817us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:34.858589 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:34.858589 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:34.858702 (+ 113us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:34.858760 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:34.858970 (+ 210us) server_negotiation.cc:300] Negotiation successful
0429 14:04:34.859060 (+ 90us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260429 14:04:34.859362 9636 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.823057 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:59236 (local address 127.2.197.254:40335)
0429 14:04:34.823100 (+ 43us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:34.823103 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:34.823247 (+ 144us) server_negotiation.cc:408] Connection header received
0429 14:04:34.823284 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:34.823286 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:34.823297 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:34.823350 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:34.851205 (+ 27855us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.852013 (+ 808us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:34.853423 (+ 1410us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.853630 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:34.855551 (+ 1921us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:34.855555 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:34.855555 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:34.855562 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:34.858026 (+ 2464us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:34.858677 (+ 651us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:34.858679 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:34.858679 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:34.858703 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:34.858874 (+ 171us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:34.858875 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:34.858875 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:34.858968 (+ 93us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:34.859015 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:34.859194 (+ 179us) server_negotiation.cc:300] Negotiation successful
0429 14:04:34.859273 (+ 79us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260429 14:04:34.859584 10319 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.850752 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:47002 (local address 127.2.197.253:44465)
0429 14:04:34.850802 (+ 50us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:34.850804 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:34.850954 (+ 150us) server_negotiation.cc:408] Connection header received
0429 14:04:34.851075 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:34.851077 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:34.851095 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:34.851131 (+ 36us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:34.851802 (+ 671us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.852507 (+ 705us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:34.853293 (+ 786us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.853471 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:34.854729 (+ 1258us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:34.854733 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:34.854733 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:34.854741 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:34.856756 (+ 2015us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:34.858532 (+ 1776us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:34.858533 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:34.858534 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:34.858564 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:34.858737 (+ 173us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:34.858738 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:34.858738 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:34.858841 (+ 103us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:34.858906 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:34.859371 (+ 465us) server_negotiation.cc:300] Negotiation successful
0429 14:04:34.859485 (+ 114us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260429 14:04:34.860154 10339 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.823096 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:36013 (local address 127.0.0.1:36576)
0429 14:04:34.823195 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:34.823205 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:34.823290 (+ 85us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:34.823420 (+ 130us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:34.823420 (+ 0us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:34.850883 (+ 27463us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:34.851161 (+ 278us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:34.851165 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.852055 (+ 890us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:34.852056 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:34.852608 (+ 552us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:34.852610 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.853582 (+ 972us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:34.854250 (+ 668us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:34.854255 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:34.854847 (+ 592us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:34.857450 (+ 2603us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:34.857451 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:34.857451 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:34.857643 (+ 192us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:34.857941 (+ 298us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:34.857942 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:34.857942 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:34.858198 (+ 256us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:34.858805 (+ 607us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:34.858806 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:34.858840 (+ 34us) client_negotiation.cc:770] Sending connection context
0429 14:04:34.859414 (+ 574us) client_negotiation.cc:241] Negotiation successful
0429 14:04:34.859507 (+ 93us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":62,"threads_started":1}
I20260429 14:04:34.860364 10340 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.850642 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.253:44465 (local address 127.0.0.1:47002)
0429 14:04:34.850858 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:34.850873 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:34.850987 (+ 114us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:34.851218 (+ 231us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:34.851219 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:34.851421 (+ 202us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:34.851643 (+ 222us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:34.851645 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.852611 (+ 966us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:34.852612 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:34.853188 (+ 576us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:34.853190 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.853520 (+ 330us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:34.854061 (+ 541us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:34.854067 (+ 6us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:34.854623 (+ 556us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:34.858026 (+ 3403us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:34.858027 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:34.858027 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:34.858431 (+ 404us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:34.858640 (+ 209us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:34.858640 (+ 0us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:34.858640 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:34.858666 (+ 26us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:34.858933 (+ 267us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:34.858934 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:34.858981 (+ 47us) client_negotiation.cc:770] Sending connection context
0429 14:04:34.859049 (+ 68us) client_negotiation.cc:241] Negotiation successful
0429 14:04:34.859135 (+ 86us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":110,"threads_started":1}
I20260429 14:04:34.860687 10338 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.822995 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.0.0.1:59236)
0429 14:04:34.823151 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:34.823163 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:34.823233 (+ 70us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:34.823414 (+ 181us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:34.823415 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:34.850749 (+ 27334us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:34.851072 (+ 323us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:34.851074 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.852810 (+ 1736us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:34.852810 (+ 0us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:34.853325 (+ 515us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:34.853327 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.853694 (+ 367us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:34.854458 (+ 764us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:34.854463 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:34.855179 (+ 716us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:34.858115 (+ 2936us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:34.858116 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:34.858116 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:34.858589 (+ 473us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:34.858783 (+ 194us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:34.858783 (+ 0us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:34.858784 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:34.858813 (+ 29us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:34.859042 (+ 229us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:34.859043 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:34.859092 (+ 49us) client_negotiation.cc:770] Sending connection context
0429 14:04:34.859467 (+ 375us) client_negotiation.cc:241] Negotiation successful
0429 14:04:34.859571 (+ 104us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":105,"threads_started":1}
I20260429 14:04:34.863220 10152 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.858435 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.253:44465 (local address 127.0.0.1:47016)
0429 14:04:34.858479 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:34.858489 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:34.858526 (+ 37us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:34.859931 (+ 1405us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:34.859931 (+ 0us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:34.859934 (+ 3us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:04:34.860086 (+ 152us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:34.860087 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.862043 (+ 1956us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:34.862044 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:34.862938 (+ 894us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:34.862940 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.863046 (+ 106us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:34.863050 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 14:04:34.863112 (+ 62us) client_negotiation.cc:241] Negotiation successful
0429 14:04:34.863155 (+ 43us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260429 14:04:34.863853 10342 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.858480 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:47016 (local address 127.2.197.253:44465)
0429 14:04:34.859561 (+ 1081us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:34.859563 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:34.859572 (+ 9us) server_negotiation.cc:408] Connection header received
0429 14:04:34.859594 (+ 22us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:34.859595 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:34.859623 (+ 28us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:34.859923 (+ 300us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:04:34.860178 (+ 255us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.861943 (+ 1765us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:34.863045 (+ 1102us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.863640 (+ 595us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:34.863702 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:34.863762 (+ 60us) server_negotiation.cc:300] Negotiation successful
0429 14:04:34.863805 (+ 43us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1062,"thread_start_us":75,"threads_started":1}
Apr 29 14:04:34 dist-test-slave-8n77 krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471473, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.254@KRBTEST.COM for oryx/127.2.197.252@KRBTEST.COM
I20260429 14:04:34.870829 10343 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.862929 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:36013 (local address 127.0.0.1:36580)
0429 14:04:34.863179 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:34.863190 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:34.863265 (+ 75us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:34.863447 (+ 182us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:34.863448 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:34.863637 (+ 189us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:34.863845 (+ 208us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:34.863847 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.864597 (+ 750us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:34.864598 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:34.865018 (+ 420us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:34.865020 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.865126 (+ 106us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:34.865597 (+ 471us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:34.865602 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:34.867222 (+ 1620us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:34.869508 (+ 2286us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:34.869509 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:34.869510 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:34.869792 (+ 282us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:34.869965 (+ 173us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:34.869966 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:34.869966 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:34.870067 (+ 101us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:34.870427 (+ 360us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:34.870428 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:34.870474 (+ 46us) client_negotiation.cc:770] Sending connection context
0429 14:04:34.870617 (+ 143us) client_negotiation.cc:241] Negotiation successful
0429 14:04:34.870729 (+ 112us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":95,"threads_started":1}
I20260429 14:04:34.870828 10318 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.863124 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36580 (local address 127.2.197.252:36013)
0429 14:04:34.863170 (+ 46us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:34.863172 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:34.863242 (+ 70us) server_negotiation.cc:408] Connection header received
0429 14:04:34.863346 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:34.863347 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:34.863356 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:34.863385 (+ 29us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:34.863949 (+ 564us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.864488 (+ 539us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:34.865125 (+ 637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.865312 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:34.867356 (+ 2044us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:34.867359 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:34.867359 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:34.867367 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:34.869400 (+ 2033us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:34.869867 (+ 467us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:34.869869 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:34.869869 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:34.869893 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:34.870195 (+ 302us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:34.870196 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:34.870197 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:34.870342 (+ 145us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:34.870435 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:34.870643 (+ 208us) server_negotiation.cc:300] Negotiation successful
0429 14:04:34.870728 (+ 85us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260429 14:04:34.871280 9583 catalog_manager.cc:7158] Initiating ChangeConfig request to add master 127.2.197.252:36013: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f"
member_type: NON_VOTER
last_known_addr {
host: "127.2.197.252"
port: 36013
}
attrs {
promote: true
}
}
dest_uuid: "f8a2e74b05bf42de93b558952af4f587"
cas_config_opid_index: 5
I20260429 14:04:34.871506 9583 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: NON_VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: true } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: NON_VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: true } }
I20260429 14:04:34.871852 10154 raft_consensus.cc:2955] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9782bdb83d3143c6a2f310e1c7de398f (127.2.197.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: NON_VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: true } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: NON_VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: true } } }
I20260429 14:04:34.872099 10289 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:34.872154 10207 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [term 1 LEARNER]: Refusing update from remote peer f8a2e74b05bf42de93b558952af4f587: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260429 14:04:34.872504 10153 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: NON_VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260429 14:04:34.872836 10154 catalog_manager.cc:7162] Successfully completed master ChangeConfig request to add master 127.2.197.252:36013
I20260429 14:04:34.872915 10154 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f8a2e74b05bf42de93b558952af4f587" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: NON_VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: true } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: NON_VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: true } } }
I20260429 14:04:34.872972 10154 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [sys.catalog]: This master's current role is: LEADER
I20260429 14:04:34.872987 10289 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [term 1 FOLLOWER]: Refusing update from remote peer f8a2e74b05bf42de93b558952af4f587: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260429 14:04:34.873180 10154 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: NON_VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:04:34.873162 10207 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9782bdb83d3143c6a2f310e1c7de398f (127.2.197.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: NON_VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: true } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: NON_VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: true } } }
I20260429 14:04:34.874358 10197 master.cc:562] Master@127.2.197.252:36013 shutting down...
I20260429 14:04:34.874586 10289 raft_consensus.cc:872] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 36013 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: NON_VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: NON_VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: true } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: NON_VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: true } }
W20260429 14:04:34.874650 10289 raft_consensus.cc:1579] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 36013 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: NON_VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: NON_VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: true } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: NON_VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: true } }
I20260429 14:04:34.875214 10289 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 7de05a456e054b9f83790eb862246127 (127.2.197.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: NON_VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: true } } }
I20260429 14:04:34.876696 10321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8a2e74b05bf42de93b558952af4f587. Latest consensus state: current_term: 1 leader_uuid: "f8a2e74b05bf42de93b558952af4f587" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: NON_VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: true } } }
I20260429 14:04:34.876783 10321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260429 14:04:34.876832 10321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8a2e74b05bf42de93b558952af4f587" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: NON_VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: true } } }
I20260429 14:04:34.876878 10321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260429 14:04:34.876912 10321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8a2e74b05bf42de93b558952af4f587" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: NON_VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: true } } }
I20260429 14:04:34.876955 10321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260429 14:04:34.877287 10153 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8a2e74b05bf42de93b558952af4f587" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: NON_VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: true } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: NON_VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: true } } }
I20260429 14:04:34.877369 10153 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [sys.catalog]: This master's current role is: LEADER
I20260429 14:04:34.877538 10236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8a2e74b05bf42de93b558952af4f587. Latest consensus state: current_term: 1 leader_uuid: "f8a2e74b05bf42de93b558952af4f587" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: NON_VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: true } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: NON_VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: true } } }
I20260429 14:04:34.877627 10236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [sys.catalog]: This master's current role is: LEARNER
I20260429 14:04:34.877676 10236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8a2e74b05bf42de93b558952af4f587" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: NON_VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: true } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: NON_VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: true } } }
I20260429 14:04:34.877720 10236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [sys.catalog]: This master's current role is: LEARNER
W20260429 14:04:34.877833 9571 consensus_peers.cc:597] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 -> Peer 9782bdb83d3143c6a2f310e1c7de398f (127.2.197.252:36013): Couldn't send request to peer 9782bdb83d3143c6a2f310e1c7de398f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260429 14:04:34.878116 10154 raft_consensus.cc:1064] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587: attempting to promote NON_VOTER 7de05a456e054b9f83790eb862246127 to VOTER
I20260429 14:04:34.878314 10154 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: false } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: NON_VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: true } }
I20260429 14:04:34.878720 10345 mvcc.cc:204] Tried to move back new op lower bound from 7280523156263043072 to 7280523156122419200. Current Snapshot: MvccSnapshot[applied={T|T < 7280523156137500672}]
I20260429 14:04:34.878783 10197 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260429 14:04:34.878856 10197 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260429 14:04:34.878894 10197 tablet_replica.cc:333] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f: stopping tablet replica
I20260429 14:04:34.879016 10207 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [term 1 LEARNER]: Refusing update from remote peer f8a2e74b05bf42de93b558952af4f587: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260429 14:04:34.879179 9583 master.cc:680] Remote error: unable to get registration information for peer 9782bdb83d3143c6a2f310e1c7de398f: Service unavailable: service kudu.master.MasterService not registered on Master
I20260429 14:04:34.879194 10153 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260429 14:04:34.879346 9571 consensus_peers.cc:597] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 -> Peer 9782bdb83d3143c6a2f310e1c7de398f (127.2.197.252:36013): Couldn't send request to peer 9782bdb83d3143c6a2f310e1c7de398f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260429 14:04:34.879874 10350 raft_consensus.cc:2955] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7de05a456e054b9f83790eb862246127 (127.2.197.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: false } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: NON_VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: true } } }
I20260429 14:04:34.880856 10207 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7de05a456e054b9f83790eb862246127 (127.2.197.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: false } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: NON_VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: true } } }
I20260429 14:04:34.881672 10349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8a2e74b05bf42de93b558952af4f587" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: false } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: NON_VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: true } } }
I20260429 14:04:34.881748 10349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [sys.catalog]: This master's current role is: FOLLOWER
I20260429 14:04:34.882011 10354 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8a2e74b05bf42de93b558952af4f587" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: false } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: NON_VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: true } } }
I20260429 14:04:34.882122 10354 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [sys.catalog]: This master's current role is: LEADER
I20260429 14:04:34.882014 10153 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f8a2e74b05bf42de93b558952af4f587" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: false } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: NON_VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: true } } }
I20260429 14:04:34.882150 10153 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [sys.catalog]: This master's current role is: LEADER
I20260429 14:04:34.890511 10343 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.890378 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:36013 (local address 127.0.0.1:36584)
0429 14:04:34.890436 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:34.890472 (+ 36us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.252:36013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260429 14:04:34.890668 9583 master.cc:680] Network error: unable to get registration information for peer 9782bdb83d3143c6a2f310e1c7de398f: Client connection negotiation failed: client connection to 127.2.197.252:36013: connect: Connection refused (error 111)
I20260429 14:04:34.897758 10197 master.cc:584] Master@127.2.197.252:36013 shutdown complete.
I20260429 14:04:34.897838 10197 master_runner.cc:305] Clearing existing system tablet
I20260429 14:04:34.897984 10197 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:34.898036 10197 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:34.898837 10197 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260429 14:04:34.899436 10356 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:34.899653 10197 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:34.899717 10197 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:34.899762 10197 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-2/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-2/wal
uuid: "9782bdb83d3143c6a2f310e1c7de398f"
format_stamp: "Formatted at 2026-04-29 14:04:34 on dist-test-slave-8n77"
server_key: "15e070571f9d9bb7e6242232ff6f415e"
server_key_iv: "3eeaeeb76e14b7201fc2019860c1b371"
server_key_version: "encryptionkey@0"
I20260429 14:04:34.900365 10197 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:04:34.902042 10197 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260429 14:04:34.902104 10197 log.cc:1199] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260429 14:04:34.902384 10197 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f: Deleting consensus metadata
I20260429 14:04:34.902511 10197 master_runner.cc:315] Copying system tablet from 127.2.197.254:40335
I20260429 14:04:34.903221 10197 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f: tablet copy: Beginning tablet copy session from remote peer at address 127.2.197.254:40335
I20260429 14:04:34.908600 10343 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.908470 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:36013 (local address 127.0.0.1:36592)
0429 14:04:34.908519 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:34.908548 (+ 29us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.252:36013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260429 14:04:34.908728 9583 master.cc:680] Network error: unable to get registration information for peer 9782bdb83d3143c6a2f310e1c7de398f: Client connection negotiation failed: client connection to 127.2.197.252:36013: connect: Connection refused (error 111)
I20260429 14:04:34.909834 10362 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.903580 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.0.0.1:59250)
0429 14:04:34.903795 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:34.903805 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:34.903894 (+ 89us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:34.904041 (+ 147us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:34.904042 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:34.904215 (+ 173us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:34.904406 (+ 191us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:34.904408 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.905361 (+ 953us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:34.905362 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:34.905774 (+ 412us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:34.905775 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.905854 (+ 79us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:34.906465 (+ 611us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:34.906470 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:34.906778 (+ 308us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:34.909018 (+ 2240us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:34.909019 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:34.909019 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:34.909169 (+ 150us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:34.909324 (+ 155us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:34.909324 (+ 0us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:34.909325 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:34.909344 (+ 19us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:34.909539 (+ 195us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:34.909540 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:34.909614 (+ 74us) client_negotiation.cc:770] Sending connection context
0429 14:04:34.909668 (+ 54us) client_negotiation.cc:241] Negotiation successful
0429 14:04:34.909750 (+ 82us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":109,"threads_started":1}
I20260429 14:04:34.909883 9636 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.903730 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:59250 (local address 127.2.197.254:40335)
0429 14:04:34.903807 (+ 77us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:34.903810 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:34.903899 (+ 89us) server_negotiation.cc:408] Connection header received
0429 14:04:34.903960 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:34.903962 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:34.903973 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:34.904030 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:34.904483 (+ 453us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.905267 (+ 784us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:34.905863 (+ 596us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:34.906043 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:34.906894 (+ 851us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:34.906898 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:34.906898 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:34.906905 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:34.908935 (+ 2030us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:34.909241 (+ 306us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:34.909242 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:34.909242 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:34.909267 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:34.909376 (+ 109us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:34.909376 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:34.909377 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:34.909483 (+ 106us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:34.909554 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:34.909714 (+ 160us) server_negotiation.cc:300] Negotiation successful
0429 14:04:34.909794 (+ 80us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260429 14:04:34.910122 9606 tablet_copy_service.cc:140] P f8a2e74b05bf42de93b558952af4f587: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9782bdb83d3143c6a2f310e1c7de398f ({username='oryx', principal='oryx/127.2.197.252@KRBTEST.COM'} at 127.0.0.1:59250)
I20260429 14:04:34.910169 9606 tablet_copy_service.cc:161] P f8a2e74b05bf42de93b558952af4f587: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9782bdb83d3143c6a2f310e1c7de398f at {username='oryx', principal='oryx/127.2.197.252@KRBTEST.COM'} at 127.0.0.1:59250: session id = 9782bdb83d3143c6a2f310e1c7de398f-00000000000000000000000000000000
I20260429 14:04:34.910787 9606 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587: Tablet Copy: opened 0 blocks and 1 log segments
I20260429 14:04:34.911247 10197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260429 14:04:34.912217 10197 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f: tablet copy: Starting download of 0 data blocks...
I20260429 14:04:34.912370 10197 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f: tablet copy: Starting download of 1 WAL segments...
I20260429 14:04:34.913358 10197 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260429 14:04:34.914268 9606 tablet_copy_service.cc:342] P f8a2e74b05bf42de93b558952af4f587: Request end of tablet copy session 9782bdb83d3143c6a2f310e1c7de398f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.197.252@KRBTEST.COM'} at 127.0.0.1:59250
I20260429 14:04:34.914400 9606 tablet_copy_service.cc:434] P f8a2e74b05bf42de93b558952af4f587: ending tablet copy session 9782bdb83d3143c6a2f310e1c7de398f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9782bdb83d3143c6a2f310e1c7de398f
W20260429 14:04:34.917259 10197 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260429 14:04:34.918133 10197 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:34.919574 10363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:34.919667 10366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:34.919727 10364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:34.919850 10197 server_base.cc:1061] running on GCE node
I20260429 14:04:34.920022 10197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:34.920209 10197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:34.921339 10197 hybrid_clock.cc:648] HybridClock initialized: now 1777471474921326 us; error 29 us; skew 500 ppm
Apr 29 14:04:34 dist-test-slave-8n77 krb5kdc[9538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471474, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:34.922986 10197 init.cc:377] Logged in from keytab as oryx/127.2.197.252@KRBTEST.COM (short username oryx)
I20260429 14:04:34.923291 10197 webserver.cc:492] Webserver started at http://127.2.197.252:42781/ using document root <none> and password file <none>
I20260429 14:04:34.923411 10197 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:34.923447 10197 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:34.924279 10197 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:34.924964 10373 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:34.925184 10197 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:34.925241 10197 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-2/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-2/wal
uuid: "9782bdb83d3143c6a2f310e1c7de398f"
format_stamp: "Formatted at 2026-04-29 14:04:34 on dist-test-slave-8n77"
server_key: "15e070571f9d9bb7e6242232ff6f415e"
server_key_iv: "3eeaeeb76e14b7201fc2019860c1b371"
server_key_version: "encryptionkey@0"
I20260429 14:04:34.925294 10197 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:34.929397 9779 heartbeater.cc:499] Master 127.2.197.254:40335 was elected leader, sending a full tablet report...
I20260429 14:04:34.943063 10343 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:34.942934 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:36013 (local address 127.0.0.1:36596)
0429 14:04:34.942979 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:34.943013 (+ 34us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.252:36013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260429 14:04:34.943209 9583 master.cc:680] Network error: unable to get registration information for peer 9782bdb83d3143c6a2f310e1c7de398f: Client connection negotiation failed: client connection to 127.2.197.252:36013: connect: Connection refused (error 111)
I20260429 14:04:34.966907 10197 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:34.968722 10197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:34.972735 10425 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.252:36013 every 8 connection(s)
I20260429 14:04:34.972747 10197 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.252:36013
I20260429 14:04:34.973176 10197 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/master-2/data/info.pb
I20260429 14:04:34.974020 10426 sys_catalog.cc:263] Verifying existing consensus state
I20260429 14:04:34.974538 10426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f: Bootstrap starting.
I20260429 14:04:34.976678 10426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260429 14:04:34.976915 10426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f: Bootstrap complete.
I20260429 14:04:34.977171 10426 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: false } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: NON_VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: true } }
I20260429 14:04:34.977242 10426 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9782bdb83d3143c6a2f310e1c7de398f, State: Initialized, Role: LEARNER
I20260429 14:04:34.977268 10426 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: false } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: NON_VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: true } }
I20260429 14:04:34.977475 10426 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:04:34.977492 10429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: false } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: NON_VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: true } } }
I20260429 14:04:34.977587 10429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [sys.catalog]: This master's current role is: LEARNER
I20260429 14:04:34.979379 10440 catalog_manager.cc:1269] Loaded cluster ID: bfb901df49664bdeb1e28c49a5039180
I20260429 14:04:34.979450 10440 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f: loading cluster ID for follower catalog manager: success
I20260429 14:04:34.980427 10440 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f: acquiring CA information for follower catalog manager: success
I20260429 14:04:34.980707 10440 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260429 14:04:35.012894 10343 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:35.010024 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:36013 (local address 127.0.0.1:36602)
0429 14:04:35.010114 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:35.010130 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:35.010190 (+ 60us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:35.010557 (+ 367us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:35.010559 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:35.010563 (+ 4us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:04:35.010868 (+ 305us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:35.010874 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:35.011987 (+ 1113us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:35.011988 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:35.012622 (+ 634us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:35.012625 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:35.012724 (+ 99us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:35.012730 (+ 6us) client_negotiation.cc:770] Sending connection context
0429 14:04:35.012762 (+ 32us) client_negotiation.cc:241] Negotiation successful
0429 14:04:35.012810 (+ 48us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260429 14:04:35.013444 10441 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:35.010152 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36602 (local address 127.2.197.252:36013)
0429 14:04:35.010381 (+ 229us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:35.010383 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:35.010394 (+ 11us) server_negotiation.cc:408] Connection header received
0429 14:04:35.010424 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:35.010424 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:35.010451 (+ 27us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:35.010549 (+ 98us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:04:35.010984 (+ 435us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:35.011862 (+ 878us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:35.012728 (+ 866us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:35.013221 (+ 493us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:35.013326 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:35.013348 (+ 22us) server_negotiation.cc:300] Negotiation successful
0429 14:04:35.013390 (+ 42us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":82,"threads_started":1}
I20260429 14:04:35.125877 9915 heartbeater.cc:499] Master 127.2.197.254:40335 was elected leader, sending a full tablet report...
I20260429 14:04:35.261602 10400 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260429 14:04:35.261782 10429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8a2e74b05bf42de93b558952af4f587. Latest consensus state: current_term: 1 leader_uuid: "f8a2e74b05bf42de93b558952af4f587" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: false } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: NON_VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: true } } }
I20260429 14:04:35.261915 10429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [sys.catalog]: This master's current role is: LEARNER
I20260429 14:04:35.314368 10051 heartbeater.cc:499] Master 127.2.197.254:40335 was elected leader, sending a full tablet report...
I20260429 14:04:35.795588 10153 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8a2e74b05bf42de93b558952af4f587" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: false } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: NON_VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: true } } }
I20260429 14:04:35.795742 10153 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [sys.catalog]: This master's current role is: LEADER
I20260429 14:04:35.796190 10443 raft_consensus.cc:1064] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587: attempting to promote NON_VOTER 9782bdb83d3143c6a2f310e1c7de398f to VOTER
I20260429 14:04:35.796337 10443 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: false } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: false } }
I20260429 14:04:35.796931 10400 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [term 1 LEARNER]: Refusing update from remote peer f8a2e74b05bf42de93b558952af4f587: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260429 14:04:35.796972 10207 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [term 1 FOLLOWER]: Refusing update from remote peer f8a2e74b05bf42de93b558952af4f587: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260429 14:04:35.797204 10442 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260429 14:04:35.797708 10153 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260429 14:04:35.798702 10442 raft_consensus.cc:2955] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9782bdb83d3143c6a2f310e1c7de398f (127.2.197.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: false } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: false } } }
I20260429 14:04:35.798908 10207 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9782bdb83d3143c6a2f310e1c7de398f (127.2.197.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: false } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: false } } }
I20260429 14:04:35.798908 10400 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9782bdb83d3143c6a2f310e1c7de398f (127.2.197.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: false } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: false } } }
I20260429 14:04:35.799599 10442 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f8a2e74b05bf42de93b558952af4f587" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: false } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: false } } }
I20260429 14:04:35.799678 10442 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [sys.catalog]: This master's current role is: LEADER
I20260429 14:04:35.799759 10442 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8a2e74b05bf42de93b558952af4f587" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: false } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: false } } }
I20260429 14:04:35.799814 10442 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [sys.catalog]: This master's current role is: LEADER
I20260429 14:04:35.799867 10442 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8a2e74b05bf42de93b558952af4f587" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: false } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: false } } }
I20260429 14:04:35.799917 10442 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587 [sys.catalog]: This master's current role is: LEADER
I20260429 14:04:35.800019 10446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8a2e74b05bf42de93b558952af4f587" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: false } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: false } } }
I20260429 14:04:35.800100 10446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [sys.catalog]: This master's current role is: FOLLOWER
I20260429 14:04:35.800190 10447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8a2e74b05bf42de93b558952af4f587" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: false } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: false } } }
I20260429 14:04:35.800258 10447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [sys.catalog]: This master's current role is: FOLLOWER
I20260429 14:04:35.924998 10449 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:35.918424 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:47032 (local address 127.2.197.253:44465)
0429 14:04:35.918604 (+ 180us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:35.918606 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:35.918614 (+ 8us) server_negotiation.cc:408] Connection header received
0429 14:04:35.918636 (+ 22us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:35.918637 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:35.918647 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:35.918708 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:35.919748 (+ 1040us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:35.920397 (+ 649us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:35.920991 (+ 594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:35.921127 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:35.922254 (+ 1127us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:35.922258 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:35.922258 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:35.922264 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:35.923917 (+ 1653us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:35.924248 (+ 331us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:35.924249 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:35.924250 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:35.924276 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:35.924428 (+ 152us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:35.924429 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:35.924429 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:35.924553 (+ 124us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:35.924642 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:35.924831 (+ 189us) server_negotiation.cc:300] Negotiation successful
0429 14:04:35.924919 (+ 88us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":129,"threads_started":1}
I20260429 14:04:35.927656 10450 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:35.925491 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.0.0.1:59256)
0429 14:04:35.925662 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:35.925680 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:35.925736 (+ 56us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:35.925916 (+ 180us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:35.925917 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:35.925919 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:04:35.926059 (+ 140us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:35.926060 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:35.926814 (+ 754us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:35.926816 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:35.927439 (+ 623us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:35.927441 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:35.927516 (+ 75us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:35.927530 (+ 14us) client_negotiation.cc:770] Sending connection context
0429 14:04:35.927552 (+ 22us) client_negotiation.cc:241] Negotiation successful
0429 14:04:35.927601 (+ 49us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":95,"threads_started":1}
I20260429 14:04:35.928249 10451 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:35.925599 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:59256 (local address 127.2.197.254:40335)
0429 14:04:35.925720 (+ 121us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:35.925723 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:35.925740 (+ 17us) server_negotiation.cc:408] Connection header received
0429 14:04:35.925815 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:35.925816 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:35.925825 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:35.925907 (+ 82us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:04:35.926135 (+ 228us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:35.926726 (+ 591us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:35.927701 (+ 975us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:35.928139 (+ 438us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:35.928146 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:35.928167 (+ 21us) server_negotiation.cc:300] Negotiation successful
0429 14:04:35.928191 (+ 24us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":76,"threads_started":1}
I20260429 14:04:35.930807 10450 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:35.928964 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.253:44465 (local address 127.0.0.1:47046)
0429 14:04:35.929039 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:35.929051 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:35.929101 (+ 50us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:35.929213 (+ 112us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:35.929214 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:35.929217 (+ 3us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:04:35.929329 (+ 112us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:35.929330 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:35.929917 (+ 587us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:35.929917 (+ 0us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:35.930582 (+ 665us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:35.930584 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:35.930647 (+ 63us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:35.930651 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 14:04:35.930686 (+ 35us) client_negotiation.cc:241] Negotiation successful
0429 14:04:35.930741 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260429 14:04:35.931231 10449 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:35.929049 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:47046 (local address 127.2.197.253:44465)
0429 14:04:35.929101 (+ 52us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:35.929102 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:35.929110 (+ 8us) server_negotiation.cc:408] Connection header received
0429 14:04:35.929145 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:35.929146 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:35.929154 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:35.929201 (+ 47us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:04:35.929385 (+ 184us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:35.929854 (+ 469us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:35.930686 (+ 832us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:35.931144 (+ 458us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:35.931150 (+ 6us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:35.931169 (+ 19us) server_negotiation.cc:300] Negotiation successful
0429 14:04:35.931193 (+ 24us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260429 14:04:35.933981 10450 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:35.931933 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:36013 (local address 127.0.0.1:36606)
0429 14:04:35.931984 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:35.931995 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:35.932038 (+ 43us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:35.932458 (+ 420us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:35.932463 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:35.932465 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:04:35.932583 (+ 118us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:35.932584 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:35.933227 (+ 643us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:35.933228 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:35.933767 (+ 539us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:35.933769 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:35.933877 (+ 108us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:35.933880 (+ 3us) client_negotiation.cc:770] Sending connection context
0429 14:04:35.933904 (+ 24us) client_negotiation.cc:241] Negotiation successful
0429 14:04:35.933929 (+ 25us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260429 14:04:35.934533 10452 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:35.932059 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36606 (local address 127.2.197.252:36013)
0429 14:04:35.932208 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:35.932210 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:35.932217 (+ 7us) server_negotiation.cc:408] Connection header received
0429 14:04:35.932237 (+ 20us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:35.932238 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:35.932246 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:35.932432 (+ 186us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:04:35.932661 (+ 229us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:35.933136 (+ 475us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:35.933923 (+ 787us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:35.934413 (+ 490us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:35.934421 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:35.934441 (+ 20us) server_negotiation.cc:300] Negotiation successful
0429 14:04:35.934466 (+ 25us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":79,"threads_started":1}
I20260429 14:04:35.941013 10452 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:35.935603 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36620 (local address 127.2.197.252:36013)
0429 14:04:35.935663 (+ 60us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:35.935665 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:35.935673 (+ 8us) server_negotiation.cc:408] Connection header received
0429 14:04:35.935694 (+ 21us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:35.935695 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:35.935705 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:35.935776 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:35.936401 (+ 625us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:35.936853 (+ 452us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:35.937378 (+ 525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:35.937569 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:35.938597 (+ 1028us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:35.938601 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:35.938601 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:35.938607 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:35.939984 (+ 1377us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:35.940327 (+ 343us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:35.940328 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:35.940329 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:35.940353 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:35.940517 (+ 164us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:35.940518 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:35.940518 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:35.940612 (+ 94us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:35.940722 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:35.940821 (+ 99us) server_negotiation.cc:300] Negotiation successful
0429 14:04:35.940912 (+ 91us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260429 14:04:35.943643 10453 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:35.941559 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.0.0.1:59258)
0429 14:04:35.941693 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:35.941706 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:35.941774 (+ 68us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:35.941880 (+ 106us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:35.941881 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:35.941883 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:04:35.942102 (+ 219us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:35.942103 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:35.942929 (+ 826us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:35.942930 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:35.943478 (+ 548us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:35.943479 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:35.943543 (+ 64us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:35.943546 (+ 3us) client_negotiation.cc:770] Sending connection context
0429 14:04:35.943565 (+ 19us) client_negotiation.cc:241] Negotiation successful
0429 14:04:35.943593 (+ 28us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":98,"threads_started":1}
I20260429 14:04:35.944207 10451 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:35.941632 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:59258 (local address 127.2.197.254:40335)
0429 14:04:35.941701 (+ 69us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:35.941703 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:35.941777 (+ 74us) server_negotiation.cc:408] Connection header received
0429 14:04:35.941815 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:35.941816 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:35.941824 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:35.941877 (+ 53us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:04:35.942206 (+ 329us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:35.942835 (+ 629us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:35.943588 (+ 753us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:35.944108 (+ 520us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:35.944115 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:35.944135 (+ 20us) server_negotiation.cc:300] Negotiation successful
0429 14:04:35.944162 (+ 27us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260429 14:04:35.946807 10453 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:35.944940 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.253:44465 (local address 127.0.0.1:47050)
0429 14:04:35.944986 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:35.944996 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:35.945034 (+ 38us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:35.945219 (+ 185us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:35.945219 (+ 0us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:35.945223 (+ 4us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:04:35.945348 (+ 125us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:35.945349 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:35.945965 (+ 616us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:35.945966 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:35.946586 (+ 620us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:35.946588 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:35.946673 (+ 85us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:35.946676 (+ 3us) client_negotiation.cc:770] Sending connection context
0429 14:04:35.946708 (+ 32us) client_negotiation.cc:241] Negotiation successful
0429 14:04:35.946741 (+ 33us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260429 14:04:35.947192 10449 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:35.945040 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:47050 (local address 127.2.197.253:44465)
0429 14:04:35.945100 (+ 60us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:35.945102 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:35.945109 (+ 7us) server_negotiation.cc:408] Connection header received
0429 14:04:35.945131 (+ 22us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:35.945132 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:35.945147 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:35.945216 (+ 69us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:04:35.945417 (+ 201us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:35.945885 (+ 468us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:35.946670 (+ 785us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:35.947100 (+ 430us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:35.947106 (+ 6us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:35.947126 (+ 20us) server_negotiation.cc:300] Negotiation successful
0429 14:04:35.947150 (+ 24us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260429 14:04:35.949581 10453 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:35.947924 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:36013 (local address 127.0.0.1:36622)
0429 14:04:35.947988 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:35.948000 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:35.948046 (+ 46us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:35.948183 (+ 137us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:35.948183 (+ 0us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:35.948185 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:04:35.948291 (+ 106us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:35.948292 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:35.948885 (+ 593us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:35.948886 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:35.949418 (+ 532us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:35.949420 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:35.949486 (+ 66us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:35.949488 (+ 2us) client_negotiation.cc:770] Sending connection context
0429 14:04:35.949511 (+ 23us) client_negotiation.cc:241] Negotiation successful
0429 14:04:35.949534 (+ 23us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260429 14:04:35.950078 10452 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:35.948020 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36622 (local address 127.2.197.252:36013)
0429 14:04:35.948083 (+ 63us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:35.948085 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:35.948100 (+ 15us) server_negotiation.cc:408] Connection header received
0429 14:04:35.948117 (+ 17us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:35.948117 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:35.948124 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:35.948172 (+ 48us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:04:35.948347 (+ 175us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:35.948810 (+ 463us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:35.949514 (+ 704us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:35.949994 (+ 480us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:35.950001 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:35.950020 (+ 19us) server_negotiation.cc:300] Negotiation successful
0429 14:04:35.950041 (+ 21us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260429 14:04:35.950919 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 9650
I20260429 14:04:35.956564 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.193:45535
--local_ip_for_outbound_sockets=127.2.197.193
--tserver_master_addrs=127.2.197.254:40335
--webserver_port=43183
--webserver_interface=127.2.197.193
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.197.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:34323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.197.254:40335,127.2.197.253:44465
--tserver_master_addrs=127.2.197.254:40335,127.2.197.253:44465,127.2.197.252:36013 with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:04:36.035197 10454 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:36.035378 10454 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:36.035405 10454 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:36.036871 10454 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:36.036913 10454 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:36.036939 10454 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.193
I20260429 14:04:36.038583 10454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:34323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.197.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.193:45535
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.197.193
--webserver_port=43183
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:40335,127.2.197.253:44465,127.2.197.252:36013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.193
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:36.038771 10454 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:36.038915 10454 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:36.041436 10459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:36.041522 10454 server_base.cc:1061] running on GCE node
W20260429 14:04:36.041438 10462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:36.041636 10460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:36.041853 10454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:36.042068 10454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:36.043218 10454 hybrid_clock.cc:648] HybridClock initialized: now 1777471476043220 us; error 47 us; skew 500 ppm
Apr 29 14:04:36 dist-test-slave-8n77 krb5kdc[9538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471476, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:36.045670 10454 init.cc:377] Logged in from keytab as oryx/127.2.197.193@KRBTEST.COM (short username oryx)
I20260429 14:04:36.046236 10454 webserver.cc:492] Webserver started at http://127.2.197.193:43183/ using document root <none> and password file <none>
I20260429 14:04:36.046475 10454 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:36.046550 10454 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:36.047824 10454 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:36.048529 10469 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:36.048830 10454 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:36.048935 10454 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-0/wal
uuid: "468ecc83d41743dc981739ecb9b20067"
format_stamp: "Formatted at 2026-04-29 14:04:33 on dist-test-slave-8n77"
server_key: "114e5cdd04bc80c925a8fe57de0b8972"
server_key_iv: "b57daf51b0cfce9932059690d6e0ce44"
server_key_version: "encryptionkey@0"
I20260429 14:04:36.049234 10454 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:36.073777 10454 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:36.075913 10454 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:36.076074 10454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:36.076328 10454 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:04:36.076651 10454 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:04:36.076712 10454 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:36.076743 10454 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:04:36.076761 10454 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:36.083304 10454 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.193:45535
I20260429 14:04:36.083400 10582 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.193:45535 every 8 connection(s)
I20260429 14:04:36.083719 10454 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
Apr 29 14:04:36 dist-test-slave-8n77 krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471476, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.252@KRBTEST.COM
Apr 29 14:04:36 dist-test-slave-8n77 krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471476, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM
Apr 29 14:04:36 dist-test-slave-8n77 krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471476, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.253@KRBTEST.COM
I20260429 14:04:36.091396 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 10454
I20260429 14:04:36.091557 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 9786
I20260429 14:04:36.093480 10451 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:36.084917 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:59967 (local address 127.2.197.254:40335)
0429 14:04:36.084973 (+ 56us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:36.084975 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:36.084986 (+ 11us) server_negotiation.cc:408] Connection header received
0429 14:04:36.085422 (+ 436us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:36.085423 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:36.085436 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:36.085469 (+ 33us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:36.086458 (+ 989us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.087149 (+ 691us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:36.087860 (+ 711us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.088046 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:36.090450 (+ 2404us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:36.090454 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:36.090455 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:36.090461 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:36.092153 (+ 1692us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:36.092616 (+ 463us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:36.092617 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:36.092617 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:36.092641 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:36.092846 (+ 205us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:36.092847 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:36.092848 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:36.092953 (+ 105us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:36.093029 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:36.093285 (+ 256us) server_negotiation.cc:300] Negotiation successful
0429 14:04:36.093386 (+ 101us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260429 14:04:36.093571 10589 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:36.084791 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.2.197.193:59967)
0429 14:04:36.084918 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:36.084928 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:36.085306 (+ 378us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:36.085610 (+ 304us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:36.085615 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:36.085992 (+ 377us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:36.086354 (+ 362us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:36.086357 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.087232 (+ 875us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:36.087233 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:36.087763 (+ 530us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:36.087765 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.088117 (+ 352us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:36.088568 (+ 451us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:36.088572 (+ 4us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:36.090346 (+ 1774us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:36.092241 (+ 1895us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:36.092246 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:36.092248 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:36.092532 (+ 284us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:36.092707 (+ 175us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:36.092707 (+ 0us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:36.092708 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:36.092774 (+ 66us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:36.093018 (+ 244us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:36.093023 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:36.093153 (+ 130us) client_negotiation.cc:770] Sending connection context
0429 14:04:36.093264 (+ 111us) client_negotiation.cc:241] Negotiation successful
0429 14:04:36.093389 (+ 125us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
I20260429 14:04:36.094030 10584 heartbeater.cc:344] Connected to a master server at 127.2.197.254:40335
I20260429 14:04:36.094122 10584 heartbeater.cc:461] Registering TS with master...
I20260429 14:04:36.094355 10584 heartbeater.cc:507] Master 127.2.197.254:40335 requested a full tablet report, sending...
I20260429 14:04:36.094754 9583 ts_manager.cc:194] Re-registered known tserver with Master: 468ecc83d41743dc981739ecb9b20067 (127.2.197.193:45535)
I20260429 14:04:36.094937 10449 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:36.084501 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:49289 (local address 127.2.197.253:44465)
0429 14:04:36.084572 (+ 71us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:36.084575 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:36.084777 (+ 202us) server_negotiation.cc:408] Connection header received
0429 14:04:36.085508 (+ 731us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:36.085509 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:36.085522 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:36.085586 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:36.086417 (+ 831us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.087015 (+ 598us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:36.087757 (+ 742us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.087922 (+ 165us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:36.091452 (+ 3530us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:36.091457 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:36.091458 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:36.091465 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:36.093779 (+ 2314us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:36.094170 (+ 391us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:36.094171 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:36.094171 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:36.094196 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:36.094368 (+ 172us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:36.094369 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:36.094370 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:36.094474 (+ 104us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:36.094552 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:36.094698 (+ 146us) server_negotiation.cc:300] Negotiation successful
0429 14:04:36.094799 (+ 101us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260429 14:04:36.095142 9583 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.197.193@KRBTEST.COM'} at 127.2.197.193:59967
I20260429 14:04:36.095420 10587 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:36.084443 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.253:44465 (local address 127.2.197.193:49289)
0429 14:04:36.084654 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:36.084687 (+ 33us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:36.085254 (+ 567us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:36.085611 (+ 357us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:36.085616 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:36.085977 (+ 361us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:36.086241 (+ 264us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:36.086244 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.087106 (+ 862us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:36.087107 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:36.087677 (+ 570us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:36.087678 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.087796 (+ 118us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:36.088413 (+ 617us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:36.088423 (+ 10us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:36.090940 (+ 2517us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:36.093898 (+ 2958us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:36.093899 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:36.093900 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:36.094083 (+ 183us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:36.094258 (+ 175us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:36.094258 (+ 0us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:36.094259 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:36.094315 (+ 56us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:36.094541 (+ 226us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:36.094542 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:36.094587 (+ 45us) client_negotiation.cc:770] Sending connection context
0429 14:04:36.094647 (+ 60us) client_negotiation.cc:241] Negotiation successful
0429 14:04:36.095336 (+ 689us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":153,"threads_started":1}
I20260429 14:04:36.095746 10585 heartbeater.cc:344] Connected to a master server at 127.2.197.253:44465
I20260429 14:04:36.095798 10585 heartbeater.cc:461] Registering TS with master...
I20260429 14:04:36.095861 10585 heartbeater.cc:507] Master 127.2.197.253:44465 requested a full tablet report, sending...
I20260429 14:04:36.096182 10195 ts_manager.cc:194] Registered new tserver with Master: 468ecc83d41743dc981739ecb9b20067 (127.2.197.193:45535)
I20260429 14:04:36.099069 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.194:44633
--local_ip_for_outbound_sockets=127.2.197.194
--tserver_master_addrs=127.2.197.254:40335
--webserver_port=41023
--webserver_interface=127.2.197.194
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.197.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:34323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.197.254:40335,127.2.197.253:44465
--tserver_master_addrs=127.2.197.254:40335,127.2.197.253:44465,127.2.197.252:36013 with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260429 14:04:36.099977 10452 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:36.084442 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:60283 (local address 127.2.197.252:36013)
0429 14:04:36.084524 (+ 82us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:36.084527 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:36.084776 (+ 249us) server_negotiation.cc:408] Connection header received
0429 14:04:36.085422 (+ 646us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:36.085423 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:36.085436 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:36.085478 (+ 42us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:36.086850 (+ 1372us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.087415 (+ 565us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:36.088120 (+ 705us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.088303 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:36.090168 (+ 1865us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:36.090172 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:36.090172 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:36.090181 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:36.092150 (+ 1969us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:36.099320 (+ 7170us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:36.099322 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:36.099322 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:36.099363 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:36.099540 (+ 177us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:36.099541 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:36.099541 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:36.099630 (+ 89us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:36.099681 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:36.099834 (+ 153us) server_negotiation.cc:300] Negotiation successful
0429 14:04:36.099916 (+ 82us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260429 14:04:36.100100 10588 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:36.084443 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:36013 (local address 127.2.197.193:60283)
0429 14:04:36.084694 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:36.084705 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:36.085304 (+ 599us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:36.085610 (+ 306us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:36.085615 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:36.086481 (+ 866us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:36.086757 (+ 276us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:36.086759 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.087508 (+ 749us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:36.087509 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:36.088032 (+ 523us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:36.088034 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.088110 (+ 76us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:36.088633 (+ 523us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:36.088637 (+ 4us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:36.090003 (+ 1366us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:36.098929 (+ 8926us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:36.098930 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:36.098931 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:36.099140 (+ 209us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:36.099435 (+ 295us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:36.099436 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:36.099437 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:36.099475 (+ 38us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:36.099687 (+ 212us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:36.099688 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:36.099738 (+ 50us) client_negotiation.cc:770] Sending connection context
0429 14:04:36.099809 (+ 71us) client_negotiation.cc:241] Negotiation successful
0429 14:04:36.099946 (+ 137us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":6,"thread_start_us":134,"threads_started":1}
I20260429 14:04:36.100638 10583 heartbeater.cc:344] Connected to a master server at 127.2.197.252:36013
I20260429 14:04:36.100729 10583 heartbeater.cc:461] Registering TS with master...
I20260429 14:04:36.100806 10583 heartbeater.cc:507] Master 127.2.197.252:36013 requested a full tablet report, sending...
I20260429 14:04:36.101207 10390 ts_manager.cc:194] Registered new tserver with Master: 468ecc83d41743dc981739ecb9b20067 (127.2.197.193:45535)
W20260429 14:04:36.181074 10590 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:36.181253 10590 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:36.181278 10590 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:36.182837 10590 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:36.182885 10590 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:36.182914 10590 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.194
I20260429 14:04:36.184628 10590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:34323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.197.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.194:44633
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.197.194
--webserver_port=41023
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:40335,127.2.197.253:44465,127.2.197.252:36013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.194
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:36.184871 10590 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:36.185078 10590 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:36.187996 10596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:36.187973 10595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:36.188022 10590 server_base.cc:1061] running on GCE node
W20260429 14:04:36.187978 10598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:36.188366 10590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:36.188545 10590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:36.189807 10590 hybrid_clock.cc:648] HybridClock initialized: now 1777471476189782 us; error 37 us; skew 500 ppm
Apr 29 14:04:36 dist-test-slave-8n77 krb5kdc[9538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471476, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:36.192520 10590 init.cc:377] Logged in from keytab as oryx/127.2.197.194@KRBTEST.COM (short username oryx)
I20260429 14:04:36.193075 10590 webserver.cc:492] Webserver started at http://127.2.197.194:41023/ using document root <none> and password file <none>
I20260429 14:04:36.193271 10590 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:36.193323 10590 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:36.194689 10590 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:36.195356 10605 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:36.195673 10590 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260429 14:04:36.195739 10590 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-1/wal
uuid: "01baffeacfa6457094831efd2c408c4e"
format_stamp: "Formatted at 2026-04-29 14:04:34 on dist-test-slave-8n77"
server_key: "5b0f185c4316936dd9771540025742f9"
server_key_iv: "e8b9dfba9be31c2e4117899e0b35e2f6"
server_key_version: "encryptionkey@0"
I20260429 14:04:36.196034 10590 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:36.206574 10590 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:36.208658 10590 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:36.208766 10590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:36.208940 10590 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:04:36.209239 10590 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:04:36.209271 10590 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:36.209293 10590 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:04:36.209342 10590 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:36.215692 10590 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.194:44633
I20260429 14:04:36.215768 10718 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.194:44633 every 8 connection(s)
I20260429 14:04:36.216081 10590 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
Apr 29 14:04:36 dist-test-slave-8n77 krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471476, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.252@KRBTEST.COM
Apr 29 14:04:36 dist-test-slave-8n77 krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471476, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM
Apr 29 14:04:36 dist-test-slave-8n77 krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471476, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.253@KRBTEST.COM
I20260429 14:04:36.224049 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 10590
I20260429 14:04:36.224206 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 9922
I20260429 14:04:36.226737 10452 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:36.216923 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:51151 (local address 127.2.197.252:36013)
0429 14:04:36.216967 (+ 44us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:36.216970 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:36.217180 (+ 210us) server_negotiation.cc:408] Connection header received
0429 14:04:36.217795 (+ 615us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:36.217797 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:36.217814 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:36.217849 (+ 35us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:36.218727 (+ 878us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.219447 (+ 720us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:36.220027 (+ 580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.220210 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:36.222927 (+ 2717us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:36.222932 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:36.222933 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:36.222942 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:36.224992 (+ 2050us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:36.225523 (+ 531us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:36.225525 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:36.225525 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:36.225556 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:36.225853 (+ 297us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:36.225854 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:36.225854 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:36.225981 (+ 127us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:36.226118 (+ 137us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:36.226548 (+ 430us) server_negotiation.cc:300] Negotiation successful
0429 14:04:36.226644 (+ 96us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260429 14:04:36.226894 10724 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:36.216887 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:36013 (local address 127.2.197.194:51151)
0429 14:04:36.217087 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:36.217096 (+ 9us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:36.217665 (+ 569us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:36.217920 (+ 255us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:36.217921 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:36.218334 (+ 413us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:36.218645 (+ 311us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:36.218647 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.219555 (+ 908us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:36.219555 (+ 0us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:36.219940 (+ 385us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:36.219941 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.220075 (+ 134us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:36.220727 (+ 652us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:36.220741 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:36.222731 (+ 1990us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:36.225098 (+ 2367us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:36.225104 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:36.225107 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:36.225338 (+ 231us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:36.225700 (+ 362us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:36.225701 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:36.225702 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:36.225778 (+ 76us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:36.226105 (+ 327us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:36.226111 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:36.226254 (+ 143us) client_negotiation.cc:770] Sending connection context
0429 14:04:36.226455 (+ 201us) client_negotiation.cc:241] Negotiation successful
0429 14:04:36.226635 (+ 180us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":3,"thread_start_us":91,"threads_started":1}
I20260429 14:04:36.227658 10719 heartbeater.cc:344] Connected to a master server at 127.2.197.252:36013
I20260429 14:04:36.227944 10719 heartbeater.cc:461] Registering TS with master...
I20260429 14:04:36.228314 10719 heartbeater.cc:507] Master 127.2.197.252:36013 requested a full tablet report, sending...
I20260429 14:04:36.228780 10390 ts_manager.cc:194] Registered new tserver with Master: 01baffeacfa6457094831efd2c408c4e (127.2.197.194:44633)
I20260429 14:04:36.230657 10725 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:36.217148 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.253:44465 (local address 127.2.197.194:49419)
0429 14:04:36.217251 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:36.217262 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:36.217665 (+ 403us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:36.217904 (+ 239us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:36.217904 (+ 0us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:36.218341 (+ 437us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:36.218859 (+ 518us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:36.218860 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.219489 (+ 629us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:36.219490 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:36.220095 (+ 605us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:36.220096 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.220746 (+ 650us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:36.221362 (+ 616us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:36.221367 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:36.223908 (+ 2541us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:36.229441 (+ 5533us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:36.229442 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:36.229442 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:36.229639 (+ 197us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:36.229896 (+ 257us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:36.229897 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:36.229898 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:36.229935 (+ 37us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:36.230165 (+ 230us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:36.230166 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:36.230211 (+ 45us) client_negotiation.cc:770] Sending connection context
0429 14:04:36.230355 (+ 144us) client_negotiation.cc:241] Negotiation successful
0429 14:04:36.230490 (+ 135us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":65,"threads_started":1}
I20260429 14:04:36.230691 10449 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:36.217371 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:49419 (local address 127.2.197.253:44465)
0429 14:04:36.217447 (+ 76us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:36.217449 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:36.217460 (+ 11us) server_negotiation.cc:408] Connection header received
0429 14:04:36.217769 (+ 309us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:36.217771 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:36.217783 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:36.217816 (+ 33us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:36.218938 (+ 1122us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.219400 (+ 462us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:36.220195 (+ 795us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.220747 (+ 552us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:36.224026 (+ 3279us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:36.224031 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:36.224031 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:36.224040 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:36.229321 (+ 5281us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:36.229781 (+ 460us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:36.229783 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:36.229783 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:36.229828 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:36.229998 (+ 170us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:36.229999 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:36.229999 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:36.230098 (+ 99us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:36.230176 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:36.230416 (+ 240us) server_negotiation.cc:300] Negotiation successful
0429 14:04:36.230549 (+ 133us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260429 14:04:36.231104 10721 heartbeater.cc:344] Connected to a master server at 127.2.197.253:44465
I20260429 14:04:36.231194 10721 heartbeater.cc:461] Registering TS with master...
I20260429 14:04:36.231324 10721 heartbeater.cc:507] Master 127.2.197.253:44465 requested a full tablet report, sending...
I20260429 14:04:36.231647 10723 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:36.216886 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.2.197.194:34693)
0429 14:04:36.217028 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:36.217062 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:36.217636 (+ 574us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:36.217877 (+ 241us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:36.217880 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:36.218250 (+ 370us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:36.218570 (+ 320us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:36.218573 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.219522 (+ 949us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:36.219522 (+ 0us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:36.220095 (+ 573us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:36.220096 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.220557 (+ 461us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:36.221150 (+ 593us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:36.221154 (+ 4us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:36.223173 (+ 2019us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:36.230309 (+ 7136us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:36.230310 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:36.230310 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:36.230522 (+ 212us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:36.231023 (+ 501us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:36.231024 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:36.231024 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:36.231054 (+ 30us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:36.231317 (+ 263us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:36.231318 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:36.231380 (+ 62us) client_negotiation.cc:770] Sending connection context
0429 14:04:36.231444 (+ 64us) client_negotiation.cc:241] Negotiation successful
0429 14:04:36.231552 (+ 108us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260429 14:04:36.231690 10451 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:36.216909 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:34693 (local address 127.2.197.254:40335)
0429 14:04:36.216964 (+ 55us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:36.216966 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:36.217155 (+ 189us) server_negotiation.cc:408] Connection header received
0429 14:04:36.217740 (+ 585us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:36.217742 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:36.217755 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:36.217818 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:36.218669 (+ 851us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.219447 (+ 778us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:36.220196 (+ 749us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.220395 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:36.223267 (+ 2872us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:36.223271 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:36.223272 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:36.223279 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:36.230193 (+ 6914us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:36.230614 (+ 421us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:36.230615 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:36.230615 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:36.230641 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:36.231146 (+ 505us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:36.231147 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:36.231148 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:36.231266 (+ 118us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:36.231329 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:36.231489 (+ 160us) server_negotiation.cc:300] Negotiation successful
0429 14:04:36.231591 (+ 102us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260429 14:04:36.231707 10195 ts_manager.cc:194] Registered new tserver with Master: 01baffeacfa6457094831efd2c408c4e (127.2.197.194:44633)
I20260429 14:04:36.232053 10720 heartbeater.cc:344] Connected to a master server at 127.2.197.254:40335
I20260429 14:04:36.232110 10720 heartbeater.cc:461] Registering TS with master...
I20260429 14:04:36.232216 10720 heartbeater.cc:507] Master 127.2.197.254:40335 requested a full tablet report, sending...
I20260429 14:04:36.232410 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.195:42039
--local_ip_for_outbound_sockets=127.2.197.195
--tserver_master_addrs=127.2.197.254:40335
--webserver_port=46127
--webserver_interface=127.2.197.195
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.197.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:34323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.197.254:40335,127.2.197.253:44465
--tserver_master_addrs=127.2.197.254:40335,127.2.197.253:44465,127.2.197.252:36013 with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260429 14:04:36.232477 9583 ts_manager.cc:194] Re-registered known tserver with Master: 01baffeacfa6457094831efd2c408c4e (127.2.197.194:44633)
I20260429 14:04:36.232986 9583 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.197.194@KRBTEST.COM'} at 127.2.197.194:34693
W20260429 14:04:36.308346 10726 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:36.308524 10726 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:36.308548 10726 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:36.309955 10726 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:36.309998 10726 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:36.310024 10726 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.195
I20260429 14:04:36.311622 10726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:34323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.197.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.195:42039
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.197.195
--webserver_port=46127
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:40335,127.2.197.253:44465,127.2.197.252:36013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.195
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:36.311791 10726 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:36.311929 10726 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:36.314829 10735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:36.314884 10733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:36.314899 10732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:36.315034 10726 server_base.cc:1061] running on GCE node
I20260429 14:04:36.315296 10726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:36.315542 10726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:36.316712 10726 hybrid_clock.cc:648] HybridClock initialized: now 1777471476316696 us; error 43 us; skew 500 ppm
Apr 29 14:04:36 dist-test-slave-8n77 krb5kdc[9538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471476, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:36.319619 10726 init.cc:377] Logged in from keytab as oryx/127.2.197.195@KRBTEST.COM (short username oryx)
I20260429 14:04:36.320273 10726 webserver.cc:492] Webserver started at http://127.2.197.195:46127/ using document root <none> and password file <none>
I20260429 14:04:36.320525 10726 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:36.320612 10726 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:36.321899 10726 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:04:36.322685 10742 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:36.322889 10726 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260429 14:04:36.322974 10726 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-2/wal
uuid: "3e025d72db7c49159529617974c08232"
format_stamp: "Formatted at 2026-04-29 14:04:34 on dist-test-slave-8n77"
server_key: "40e0bf989a9a5e56f54349278d9f4326"
server_key_iv: "eb04b8fc89fd8b5f0ab564f9e58f6568"
server_key_version: "encryptionkey@0"
I20260429 14:04:36.323261 10726 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:36.331429 10726 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:36.333273 10726 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:36.333446 10726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:36.333719 10726 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:04:36.334043 10726 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:04:36.334100 10726 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:36.334133 10726 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:04:36.334164 10726 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:36.340279 10726 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.195:42039
I20260429 14:04:36.340354 10855 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.195:42039 every 8 connection(s)
I20260429 14:04:36.340662 10726 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
Apr 29 14:04:36 dist-test-slave-8n77 krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471476, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.195@KRBTEST.COM for oryx/127.2.197.253@KRBTEST.COM
I20260429 14:04:36.347164 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 10726
Apr 29 14:04:36 dist-test-slave-8n77 krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471476, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.195@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM
Apr 29 14:04:36 dist-test-slave-8n77 krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471476, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.195@KRBTEST.COM for oryx/127.2.197.252@KRBTEST.COM
I20260429 14:04:36.351284 10449 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:36.341745 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:38237 (local address 127.2.197.253:44465)
0429 14:04:36.341835 (+ 90us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:36.341837 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:36.341971 (+ 134us) server_negotiation.cc:408] Connection header received
0429 14:04:36.342457 (+ 486us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:36.342458 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:36.342469 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:36.342502 (+ 33us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:36.343652 (+ 1150us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.344273 (+ 621us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:36.344954 (+ 681us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.345147 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:36.347498 (+ 2351us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:36.347503 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:36.347503 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:36.347511 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:36.349691 (+ 2180us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:36.350371 (+ 680us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:36.350372 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:36.350373 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:36.350398 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:36.350606 (+ 208us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:36.350607 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:36.350608 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:36.350704 (+ 96us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:36.350767 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:36.351102 (+ 335us) server_negotiation.cc:300] Negotiation successful
0429 14:04:36.351197 (+ 95us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260429 14:04:36.351383 10451 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:36.341420 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:42247 (local address 127.2.197.254:40335)
0429 14:04:36.341465 (+ 45us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:36.341467 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:36.341681 (+ 214us) server_negotiation.cc:408] Connection header received
0429 14:04:36.342457 (+ 776us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:36.342459 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:36.342469 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:36.342502 (+ 33us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:36.343472 (+ 970us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.344010 (+ 538us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:36.344752 (+ 742us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.344926 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:36.347718 (+ 2792us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:36.347723 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:36.347723 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:36.347730 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:36.349469 (+ 1739us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:36.350375 (+ 906us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:36.350376 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:36.350376 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:36.350402 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:36.350622 (+ 220us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:36.350623 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:36.350623 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:36.350752 (+ 129us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:36.350816 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:36.351187 (+ 371us) server_negotiation.cc:300] Negotiation successful
0429 14:04:36.351273 (+ 86us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260429 14:04:36.351397 10862 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:36.341660 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.253:44465 (local address 127.2.197.195:38237)
0429 14:04:36.341891 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:36.341903 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:36.342361 (+ 458us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:36.342614 (+ 253us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:36.342615 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:36.343014 (+ 399us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:36.343563 (+ 549us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:36.343565 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.344352 (+ 787us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:36.344353 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:36.344858 (+ 505us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:36.344860 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.345194 (+ 334us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:36.345729 (+ 535us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:36.345733 (+ 4us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:36.347370 (+ 1637us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:36.349926 (+ 2556us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:36.349927 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:36.349928 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:36.350243 (+ 315us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:36.350464 (+ 221us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:36.350465 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:36.350465 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:36.350542 (+ 77us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:36.350801 (+ 259us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:36.350806 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:36.350958 (+ 152us) client_negotiation.cc:770] Sending connection context
0429 14:04:36.351061 (+ 103us) client_negotiation.cc:241] Negotiation successful
0429 14:04:36.351212 (+ 151us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":82,"threads_started":1}
I20260429 14:04:36.351791 10861 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:36.341394 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.2.197.195:42247)
0429 14:04:36.341540 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:36.341581 (+ 41us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:36.342359 (+ 778us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:36.342583 (+ 224us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:36.342584 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:36.343014 (+ 430us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:36.343359 (+ 345us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:36.343361 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.344116 (+ 755us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:36.344117 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:36.344656 (+ 539us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:36.344659 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.344976 (+ 317us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:36.345544 (+ 568us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:36.345557 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:36.347608 (+ 2051us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:36.349589 (+ 1981us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:36.349594 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:36.349596 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:36.350257 (+ 661us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:36.350478 (+ 221us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:36.350479 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:36.350479 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:36.350546 (+ 67us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:36.350844 (+ 298us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:36.350845 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:36.350958 (+ 113us) client_negotiation.cc:770] Sending connection context
0429 14:04:36.351054 (+ 96us) client_negotiation.cc:241] Negotiation successful
0429 14:04:36.351669 (+ 615us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":468,"thread_start_us":80,"threads_started":1}
I20260429 14:04:36.352038 10858 heartbeater.cc:344] Connected to a master server at 127.2.197.253:44465
I20260429 14:04:36.352123 10858 heartbeater.cc:461] Registering TS with master...
I20260429 14:04:36.352345 10858 heartbeater.cc:507] Master 127.2.197.253:44465 requested a full tablet report, sending...
I20260429 14:04:36.352934 10195 ts_manager.cc:194] Registered new tserver with Master: 3e025d72db7c49159529617974c08232 (127.2.197.195:42039)
I20260429 14:04:36.353132 10452 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:36.341509 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:57103 (local address 127.2.197.252:36013)
0429 14:04:36.341555 (+ 46us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:36.341557 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:36.341844 (+ 287us) server_negotiation.cc:408] Connection header received
0429 14:04:36.342439 (+ 595us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:36.342441 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:36.342452 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:36.342514 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:36.343346 (+ 832us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.343998 (+ 652us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:36.344742 (+ 744us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.345013 (+ 271us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:36.349920 (+ 4907us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:36.349923 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:36.349924 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:36.349932 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:36.351829 (+ 1897us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:36.352185 (+ 356us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:36.352186 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:36.352186 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:36.352209 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:36.352371 (+ 162us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:36.352372 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:36.352373 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:36.352485 (+ 112us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:36.352560 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:36.352963 (+ 403us) server_negotiation.cc:300] Negotiation successful
0429 14:04:36.353047 (+ 84us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260429 14:04:36.353084 10857 heartbeater.cc:344] Connected to a master server at 127.2.197.254:40335
I20260429 14:04:36.353569 10857 heartbeater.cc:461] Registering TS with master...
I20260429 14:04:36.353693 10860 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:36.341394 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:36013 (local address 127.2.197.195:57103)
0429 14:04:36.341769 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:36.341792 (+ 23us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:36.342345 (+ 553us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:36.342568 (+ 223us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:36.342578 (+ 10us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:36.342987 (+ 409us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:36.343245 (+ 258us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:36.343248 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.344090 (+ 842us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:36.344090 (+ 0us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:36.344664 (+ 574us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:36.344666 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.344770 (+ 104us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:36.345616 (+ 846us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:36.345621 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:36.349823 (+ 4202us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:36.351921 (+ 2098us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:36.351922 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:36.351922 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:36.352107 (+ 185us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:36.352270 (+ 163us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:36.352271 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:36.352271 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:36.352296 (+ 25us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:36.352654 (+ 358us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:36.352655 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:36.352699 (+ 44us) client_negotiation.cc:770] Sending connection context
0429 14:04:36.352768 (+ 69us) client_negotiation.cc:241] Negotiation successful
0429 14:04:36.352872 (+ 104us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":3,"thread_start_us":74,"threads_started":1}
I20260429 14:04:36.353705 10857 heartbeater.cc:507] Master 127.2.197.254:40335 requested a full tablet report, sending...
I20260429 14:04:36.354193 9583 ts_manager.cc:194] Re-registered known tserver with Master: 3e025d72db7c49159529617974c08232 (127.2.197.195:42039)
I20260429 14:04:36.354233 10856 heartbeater.cc:344] Connected to a master server at 127.2.197.252:36013
I20260429 14:04:36.354312 10856 heartbeater.cc:461] Registering TS with master...
I20260429 14:04:36.354411 10856 heartbeater.cc:507] Master 127.2.197.252:36013 requested a full tablet report, sending...
I20260429 14:04:36.354703 10390 ts_manager.cc:194] Registered new tserver with Master: 3e025d72db7c49159529617974c08232 (127.2.197.195:42039)
I20260429 14:04:36.354779 9583 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.197.195@KRBTEST.COM'} at 127.2.197.195:42247
I20260429 14:04:36.359023 10868 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:36.348463 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:59264 (local address 127.2.197.254:40335)
0429 14:04:36.348589 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:36.348591 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:36.348712 (+ 121us) server_negotiation.cc:408] Connection header received
0429 14:04:36.348812 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:36.348813 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:36.348825 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:36.348865 (+ 40us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:36.349897 (+ 1032us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.350747 (+ 850us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:36.353558 (+ 2811us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.353759 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:36.355269 (+ 1510us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:36.355272 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:36.355273 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:36.355280 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:36.357253 (+ 1973us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:36.358108 (+ 855us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:36.358109 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:36.358109 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:36.358137 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:36.358350 (+ 213us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:36.358351 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:36.358352 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:36.358470 (+ 118us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:36.358591 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:36.358798 (+ 207us) server_negotiation.cc:300] Negotiation successful
0429 14:04:36.358905 (+ 107us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":71,"threads_started":1}
I20260429 14:04:36.360910 10449 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:36.348689 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:47054 (local address 127.2.197.253:44465)
0429 14:04:36.351363 (+ 2674us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:36.351364 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:36.353136 (+ 1772us) server_negotiation.cc:408] Connection header received
0429 14:04:36.353236 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:36.353237 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:36.353248 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:36.353275 (+ 27us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:36.354554 (+ 1279us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.355241 (+ 687us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:36.355994 (+ 753us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.356178 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:36.357648 (+ 1470us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:36.357651 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:36.357652 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:36.357658 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:36.359455 (+ 1797us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:36.359818 (+ 363us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:36.359819 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:36.359820 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:36.359848 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:36.360030 (+ 182us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:36.360031 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:36.360031 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:36.360148 (+ 117us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:36.360262 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:36.360747 (+ 485us) server_negotiation.cc:300] Negotiation successful
0429 14:04:36.360835 (+ 88us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2659,"thread_start_us":57,"threads_started":1}
I20260429 14:04:36.361554 10871 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:36.352699 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36626 (local address 127.2.197.252:36013)
0429 14:04:36.352884 (+ 185us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:36.352886 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:36.354101 (+ 1215us) server_negotiation.cc:408] Connection header received
0429 14:04:36.354219 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:36.354220 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:36.354233 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:36.354299 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:36.356042 (+ 1743us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.356795 (+ 753us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:36.357539 (+ 744us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.357725 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:36.358649 (+ 924us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:36.358653 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:36.358653 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:36.358661 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:36.360571 (+ 1910us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:36.360941 (+ 370us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:36.360942 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:36.360942 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:36.360963 (+ 21us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:36.361119 (+ 156us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:36.361120 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:36.361120 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:36.361216 (+ 96us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:36.361298 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:36.361428 (+ 130us) server_negotiation.cc:300] Negotiation successful
0429 14:04:36.361492 (+ 64us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":72,"threads_started":1}
I20260429 14:04:36.360828 9583 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 14:04:36.362743 9583 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 14:04:36.369411 10881 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:36.366560 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:42039 (local address 127.0.0.1:48890)
0429 14:04:36.366688 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:36.366698 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:36.366755 (+ 57us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:36.367254 (+ 499us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:36.367255 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:36.367258 (+ 3us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:04:36.367503 (+ 245us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:36.367506 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.368405 (+ 899us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:36.368406 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:36.368996 (+ 590us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:36.368997 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.369091 (+ 94us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:36.369098 (+ 7us) client_negotiation.cc:770] Sending connection context
0429 14:04:36.369287 (+ 189us) client_negotiation.cc:241] Negotiation successful
0429 14:04:36.369331 (+ 44us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":87,"threads_started":1}
I20260429 14:04:36.369831 10885 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:36.366924 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48890 (local address 127.2.197.195:42039)
0429 14:04:36.367061 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:36.367076 (+ 15us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:36.367086 (+ 10us) server_negotiation.cc:408] Connection header received
0429 14:04:36.367129 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:36.367131 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:36.367149 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:36.367586 (+ 437us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:04:36.367649 (+ 63us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.368307 (+ 658us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:36.369111 (+ 804us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.369638 (+ 527us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:36.369696 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:36.369727 (+ 31us) server_negotiation.cc:300] Negotiation successful
0429 14:04:36.369767 (+ 40us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260429 14:04:36.369858 10883 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:36.366770 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:44633 (local address 127.0.0.1:46518)
0429 14:04:36.366878 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:36.366887 (+ 9us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:36.366928 (+ 41us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:36.367206 (+ 278us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:36.367207 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:36.367210 (+ 3us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:04:36.367488 (+ 278us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:36.367490 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.368754 (+ 1264us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:36.368755 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:36.369690 (+ 935us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:36.369692 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.369735 (+ 43us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:36.369738 (+ 3us) client_negotiation.cc:770] Sending connection context
0429 14:04:36.369761 (+ 23us) client_negotiation.cc:241] Negotiation successful
0429 14:04:36.369793 (+ 32us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":77,"threads_started":1}
I20260429 14:04:36.370321 10882 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:36.366621 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:45535 (local address 127.0.0.1:37764)
0429 14:04:36.366752 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:36.366763 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:36.366819 (+ 56us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:36.367803 (+ 984us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:36.367804 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:36.367806 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:04:36.368000 (+ 194us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:36.368002 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.369266 (+ 1264us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:36.369267 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:36.370129 (+ 862us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:36.370132 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.370175 (+ 43us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:36.370178 (+ 3us) client_negotiation.cc:770] Sending connection context
0429 14:04:36.370201 (+ 23us) client_negotiation.cc:241] Negotiation successful
0429 14:04:36.370232 (+ 31us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":90,"threads_started":1}
I20260429 14:04:36.370440 10790 tablet_service.cc:1511] Processing CreateTablet for tablet f67edf65ccbd4e53967be2b355f2faa6 (DEFAULT_TABLE table=test-table [id=3a46d96f6d014739b5f02fb90962a21d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:04:36.370766 10790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f67edf65ccbd4e53967be2b355f2faa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:36.371074 10884 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:36.366863 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:46518 (local address 127.2.197.194:44633)
0429 14:04:36.367027 (+ 164us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:36.367034 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:36.367044 (+ 10us) server_negotiation.cc:408] Connection header received
0429 14:04:36.367088 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:36.367090 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:36.367103 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:36.367187 (+ 84us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:04:36.367569 (+ 382us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.368658 (+ 1089us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:36.370165 (+ 1507us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.370884 (+ 719us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:36.370928 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:36.370959 (+ 31us) server_negotiation.cc:300] Negotiation successful
0429 14:04:36.371000 (+ 41us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":104,"threads_started":1}
I20260429 14:04:36.371156 10886 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:36.367088 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37764 (local address 127.2.197.193:45535)
0429 14:04:36.367643 (+ 555us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:36.367651 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:36.367661 (+ 10us) server_negotiation.cc:408] Connection header received
0429 14:04:36.367697 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:36.367700 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:36.367715 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:36.368074 (+ 359us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:04:36.368149 (+ 75us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.369174 (+ 1025us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:36.370473 (+ 1299us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.370970 (+ 497us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:36.371003 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:36.371027 (+ 24us) server_negotiation.cc:300] Negotiation successful
0429 14:04:36.371059 (+ 32us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":85,"threads_started":1}
I20260429 14:04:36.371667 10517 tablet_service.cc:1511] Processing CreateTablet for tablet f67edf65ccbd4e53967be2b355f2faa6 (DEFAULT_TABLE table=test-table [id=3a46d96f6d014739b5f02fb90962a21d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:04:36.371948 10517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f67edf65ccbd4e53967be2b355f2faa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:36.374150 10887 tablet_bootstrap.cc:492] T f67edf65ccbd4e53967be2b355f2faa6 P 3e025d72db7c49159529617974c08232: Bootstrap starting.
I20260429 14:04:36.374889 10887 tablet_bootstrap.cc:654] T f67edf65ccbd4e53967be2b355f2faa6 P 3e025d72db7c49159529617974c08232: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:36.374890 10653 tablet_service.cc:1511] Processing CreateTablet for tablet f67edf65ccbd4e53967be2b355f2faa6 (DEFAULT_TABLE table=test-table [id=3a46d96f6d014739b5f02fb90962a21d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:04:36.375159 10653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f67edf65ccbd4e53967be2b355f2faa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:36.375171 10887 log.cc:826] T f67edf65ccbd4e53967be2b355f2faa6 P 3e025d72db7c49159529617974c08232: Log is configured to *not* fsync() on all Append() calls
I20260429 14:04:36.375211 10888 tablet_bootstrap.cc:492] T f67edf65ccbd4e53967be2b355f2faa6 P 468ecc83d41743dc981739ecb9b20067: Bootstrap starting.
I20260429 14:04:36.375854 10888 tablet_bootstrap.cc:654] T f67edf65ccbd4e53967be2b355f2faa6 P 468ecc83d41743dc981739ecb9b20067: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:36.376051 10888 log.cc:826] T f67edf65ccbd4e53967be2b355f2faa6 P 468ecc83d41743dc981739ecb9b20067: Log is configured to *not* fsync() on all Append() calls
I20260429 14:04:36.376195 10887 tablet_bootstrap.cc:492] T f67edf65ccbd4e53967be2b355f2faa6 P 3e025d72db7c49159529617974c08232: No bootstrap required, opened a new log
I20260429 14:04:36.376268 10887 ts_tablet_manager.cc:1403] T f67edf65ccbd4e53967be2b355f2faa6 P 3e025d72db7c49159529617974c08232: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260429 14:04:36.376603 10888 tablet_bootstrap.cc:492] T f67edf65ccbd4e53967be2b355f2faa6 P 468ecc83d41743dc981739ecb9b20067: No bootstrap required, opened a new log
I20260429 14:04:36.376643 10888 ts_tablet_manager.cc:1403] T f67edf65ccbd4e53967be2b355f2faa6 P 468ecc83d41743dc981739ecb9b20067: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:36.377645 10888 raft_consensus.cc:359] T f67edf65ccbd4e53967be2b355f2faa6 P 468ecc83d41743dc981739ecb9b20067 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e025d72db7c49159529617974c08232" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 42039 } } peers { permanent_uuid: "01baffeacfa6457094831efd2c408c4e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 44633 } } peers { permanent_uuid: "468ecc83d41743dc981739ecb9b20067" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45535 } }
I20260429 14:04:36.377746 10888 raft_consensus.cc:385] T f67edf65ccbd4e53967be2b355f2faa6 P 468ecc83d41743dc981739ecb9b20067 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:36.377764 10888 raft_consensus.cc:740] T f67edf65ccbd4e53967be2b355f2faa6 P 468ecc83d41743dc981739ecb9b20067 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 468ecc83d41743dc981739ecb9b20067, State: Initialized, Role: FOLLOWER
I20260429 14:04:36.377866 10888 consensus_queue.cc:260] T f67edf65ccbd4e53967be2b355f2faa6 P 468ecc83d41743dc981739ecb9b20067 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e025d72db7c49159529617974c08232" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 42039 } } peers { permanent_uuid: "01baffeacfa6457094831efd2c408c4e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 44633 } } peers { permanent_uuid: "468ecc83d41743dc981739ecb9b20067" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45535 } }
I20260429 14:04:36.377947 10887 raft_consensus.cc:359] T f67edf65ccbd4e53967be2b355f2faa6 P 3e025d72db7c49159529617974c08232 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e025d72db7c49159529617974c08232" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 42039 } } peers { permanent_uuid: "01baffeacfa6457094831efd2c408c4e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 44633 } } peers { permanent_uuid: "468ecc83d41743dc981739ecb9b20067" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45535 } }
I20260429 14:04:36.378082 10887 raft_consensus.cc:385] T f67edf65ccbd4e53967be2b355f2faa6 P 3e025d72db7c49159529617974c08232 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:36.378108 10887 raft_consensus.cc:740] T f67edf65ccbd4e53967be2b355f2faa6 P 3e025d72db7c49159529617974c08232 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e025d72db7c49159529617974c08232, State: Initialized, Role: FOLLOWER
I20260429 14:04:36.378188 10887 consensus_queue.cc:260] T f67edf65ccbd4e53967be2b355f2faa6 P 3e025d72db7c49159529617974c08232 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e025d72db7c49159529617974c08232" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 42039 } } peers { permanent_uuid: "01baffeacfa6457094831efd2c408c4e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 44633 } } peers { permanent_uuid: "468ecc83d41743dc981739ecb9b20067" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45535 } }
I20260429 14:04:36.378114 10888 ts_tablet_manager.cc:1434] T f67edf65ccbd4e53967be2b355f2faa6 P 468ecc83d41743dc981739ecb9b20067: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260429 14:04:36.378440 10887 ts_tablet_manager.cc:1434] T f67edf65ccbd4e53967be2b355f2faa6 P 3e025d72db7c49159529617974c08232: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260429 14:04:36.378588 10857 heartbeater.cc:499] Master 127.2.197.254:40335 was elected leader, sending a full tablet report...
I20260429 14:04:36.378909 10891 tablet_bootstrap.cc:492] T f67edf65ccbd4e53967be2b355f2faa6 P 01baffeacfa6457094831efd2c408c4e: Bootstrap starting.
I20260429 14:04:36.379593 10584 heartbeater.cc:499] Master 127.2.197.254:40335 was elected leader, sending a full tablet report...
I20260429 14:04:36.380383 10891 tablet_bootstrap.cc:654] T f67edf65ccbd4e53967be2b355f2faa6 P 01baffeacfa6457094831efd2c408c4e: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:36.380801 10891 log.cc:826] T f67edf65ccbd4e53967be2b355f2faa6 P 01baffeacfa6457094831efd2c408c4e: Log is configured to *not* fsync() on all Append() calls
I20260429 14:04:36.381471 10891 tablet_bootstrap.cc:492] T f67edf65ccbd4e53967be2b355f2faa6 P 01baffeacfa6457094831efd2c408c4e: No bootstrap required, opened a new log
I20260429 14:04:36.381587 10891 ts_tablet_manager.cc:1403] T f67edf65ccbd4e53967be2b355f2faa6 P 01baffeacfa6457094831efd2c408c4e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260429 14:04:36.382927 10891 raft_consensus.cc:359] T f67edf65ccbd4e53967be2b355f2faa6 P 01baffeacfa6457094831efd2c408c4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e025d72db7c49159529617974c08232" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 42039 } } peers { permanent_uuid: "01baffeacfa6457094831efd2c408c4e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 44633 } } peers { permanent_uuid: "468ecc83d41743dc981739ecb9b20067" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45535 } }
I20260429 14:04:36.383054 10891 raft_consensus.cc:385] T f67edf65ccbd4e53967be2b355f2faa6 P 01baffeacfa6457094831efd2c408c4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:36.383100 10891 raft_consensus.cc:740] T f67edf65ccbd4e53967be2b355f2faa6 P 01baffeacfa6457094831efd2c408c4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01baffeacfa6457094831efd2c408c4e, State: Initialized, Role: FOLLOWER
I20260429 14:04:36.383190 10891 consensus_queue.cc:260] T f67edf65ccbd4e53967be2b355f2faa6 P 01baffeacfa6457094831efd2c408c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e025d72db7c49159529617974c08232" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 42039 } } peers { permanent_uuid: "01baffeacfa6457094831efd2c408c4e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 44633 } } peers { permanent_uuid: "468ecc83d41743dc981739ecb9b20067" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45535 } }
I20260429 14:04:36.383423 10891 ts_tablet_manager.cc:1434] T f67edf65ccbd4e53967be2b355f2faa6 P 01baffeacfa6457094831efd2c408c4e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260429 14:04:36.383461 10720 heartbeater.cc:499] Master 127.2.197.254:40335 was elected leader, sending a full tablet report...
W20260429 14:04:36.466758 10722 tablet.cc:2404] T f67edf65ccbd4e53967be2b355f2faa6 P 01baffeacfa6457094831efd2c408c4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 14:04:36.486677 10893 raft_consensus.cc:493] T f67edf65ccbd4e53967be2b355f2faa6 P 3e025d72db7c49159529617974c08232 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:04:36.486811 10893 raft_consensus.cc:515] T f67edf65ccbd4e53967be2b355f2faa6 P 3e025d72db7c49159529617974c08232 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e025d72db7c49159529617974c08232" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 42039 } } peers { permanent_uuid: "01baffeacfa6457094831efd2c408c4e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 44633 } } peers { permanent_uuid: "468ecc83d41743dc981739ecb9b20067" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45535 } }
I20260429 14:04:36.487197 10893 leader_election.cc:290] T f67edf65ccbd4e53967be2b355f2faa6 P 3e025d72db7c49159529617974c08232 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01baffeacfa6457094831efd2c408c4e (127.2.197.194:44633), 468ecc83d41743dc981739ecb9b20067 (127.2.197.193:45535)
I20260429 14:04:36.490192 10862 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:36.487415 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:45535 (local address 127.2.197.195:37491)
0429 14:04:36.487515 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:36.487529 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:36.487574 (+ 45us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:36.487737 (+ 163us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:36.487738 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:36.487740 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:04:36.487991 (+ 251us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:36.487993 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.489005 (+ 1012us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:36.489006 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:36.489944 (+ 938us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:36.489946 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.490027 (+ 81us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:36.490029 (+ 2us) client_negotiation.cc:770] Sending connection context
0429 14:04:36.490065 (+ 36us) client_negotiation.cc:241] Negotiation successful
0429 14:04:36.490109 (+ 44us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260429 14:04:36.490192 10860 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:36.487415 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:44633 (local address 127.2.197.195:43441)
0429 14:04:36.487522 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:36.487532 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:36.487574 (+ 42us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:36.487739 (+ 165us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:36.487740 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:36.487740 (+ 0us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:04:36.487991 (+ 251us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:36.487993 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.489005 (+ 1012us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:36.489006 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:36.489944 (+ 938us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:36.489946 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.490025 (+ 79us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:36.490029 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 14:04:36.490065 (+ 36us) client_negotiation.cc:241] Negotiation successful
0429 14:04:36.490104 (+ 39us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":55}
I20260429 14:04:36.490697 10886 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:36.487556 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:37491 (local address 127.2.197.193:45535)
0429 14:04:36.487602 (+ 46us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:36.487604 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:36.487611 (+ 7us) server_negotiation.cc:408] Connection header received
0429 14:04:36.487640 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:36.487641 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:36.487651 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:36.487717 (+ 66us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:04:36.488114 (+ 397us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.488901 (+ 787us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:36.490062 (+ 1161us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.490563 (+ 501us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:36.490572 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:36.490597 (+ 25us) server_negotiation.cc:300] Negotiation successful
0429 14:04:36.490633 (+ 36us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260429 14:04:36.490754 10884 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:36.487491 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:43441 (local address 127.2.197.194:44633)
0429 14:04:36.487543 (+ 52us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:36.487545 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:36.487558 (+ 13us) server_negotiation.cc:408] Connection header received
0429 14:04:36.487640 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:36.487641 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:36.487651 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:36.487723 (+ 72us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:04:36.488114 (+ 391us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.488895 (+ 781us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:36.490062 (+ 1167us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.490631 (+ 569us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:36.490638 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:36.490658 (+ 20us) server_negotiation.cc:300] Negotiation successful
0429 14:04:36.490690 (+ 32us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260429 14:04:36.491043 10673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f67edf65ccbd4e53967be2b355f2faa6" candidate_uuid: "3e025d72db7c49159529617974c08232" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01baffeacfa6457094831efd2c408c4e" is_pre_election: true
I20260429 14:04:36.491068 10537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f67edf65ccbd4e53967be2b355f2faa6" candidate_uuid: "3e025d72db7c49159529617974c08232" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "468ecc83d41743dc981739ecb9b20067" is_pre_election: true
I20260429 14:04:36.491199 10673 raft_consensus.cc:2468] T f67edf65ccbd4e53967be2b355f2faa6 P 01baffeacfa6457094831efd2c408c4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e025d72db7c49159529617974c08232 in term 0.
I20260429 14:04:36.491367 10537 raft_consensus.cc:2468] T f67edf65ccbd4e53967be2b355f2faa6 P 468ecc83d41743dc981739ecb9b20067 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e025d72db7c49159529617974c08232 in term 0.
I20260429 14:04:36.491489 10744 leader_election.cc:304] T f67edf65ccbd4e53967be2b355f2faa6 P 3e025d72db7c49159529617974c08232 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01baffeacfa6457094831efd2c408c4e, 3e025d72db7c49159529617974c08232; no voters:
I20260429 14:04:36.491729 10893 raft_consensus.cc:2804] T f67edf65ccbd4e53967be2b355f2faa6 P 3e025d72db7c49159529617974c08232 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 14:04:36.491813 10893 raft_consensus.cc:493] T f67edf65ccbd4e53967be2b355f2faa6 P 3e025d72db7c49159529617974c08232 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 14:04:36.491833 10893 raft_consensus.cc:3060] T f67edf65ccbd4e53967be2b355f2faa6 P 3e025d72db7c49159529617974c08232 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:36.492599 10893 raft_consensus.cc:515] T f67edf65ccbd4e53967be2b355f2faa6 P 3e025d72db7c49159529617974c08232 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e025d72db7c49159529617974c08232" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 42039 } } peers { permanent_uuid: "01baffeacfa6457094831efd2c408c4e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 44633 } } peers { permanent_uuid: "468ecc83d41743dc981739ecb9b20067" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45535 } }
I20260429 14:04:36.492833 10893 leader_election.cc:290] T f67edf65ccbd4e53967be2b355f2faa6 P 3e025d72db7c49159529617974c08232 [CANDIDATE]: Term 1 election: Requested vote from peers 01baffeacfa6457094831efd2c408c4e (127.2.197.194:44633), 468ecc83d41743dc981739ecb9b20067 (127.2.197.193:45535)
I20260429 14:04:36.492926 10537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f67edf65ccbd4e53967be2b355f2faa6" candidate_uuid: "3e025d72db7c49159529617974c08232" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "468ecc83d41743dc981739ecb9b20067"
I20260429 14:04:36.492923 10673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f67edf65ccbd4e53967be2b355f2faa6" candidate_uuid: "3e025d72db7c49159529617974c08232" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01baffeacfa6457094831efd2c408c4e"
I20260429 14:04:36.493011 10537 raft_consensus.cc:3060] T f67edf65ccbd4e53967be2b355f2faa6 P 468ecc83d41743dc981739ecb9b20067 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:36.493011 10673 raft_consensus.cc:3060] T f67edf65ccbd4e53967be2b355f2faa6 P 01baffeacfa6457094831efd2c408c4e [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:36.493840 10673 raft_consensus.cc:2468] T f67edf65ccbd4e53967be2b355f2faa6 P 01baffeacfa6457094831efd2c408c4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e025d72db7c49159529617974c08232 in term 1.
I20260429 14:04:36.493853 10537 raft_consensus.cc:2468] T f67edf65ccbd4e53967be2b355f2faa6 P 468ecc83d41743dc981739ecb9b20067 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e025d72db7c49159529617974c08232 in term 1.
I20260429 14:04:36.494122 10746 leader_election.cc:304] T f67edf65ccbd4e53967be2b355f2faa6 P 3e025d72db7c49159529617974c08232 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e025d72db7c49159529617974c08232, 468ecc83d41743dc981739ecb9b20067; no voters:
I20260429 14:04:36.494246 10893 raft_consensus.cc:2804] T f67edf65ccbd4e53967be2b355f2faa6 P 3e025d72db7c49159529617974c08232 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:04:36.494524 10893 raft_consensus.cc:697] T f67edf65ccbd4e53967be2b355f2faa6 P 3e025d72db7c49159529617974c08232 [term 1 LEADER]: Becoming Leader. State: Replica: 3e025d72db7c49159529617974c08232, State: Running, Role: LEADER
I20260429 14:04:36.494700 10893 consensus_queue.cc:237] T f67edf65ccbd4e53967be2b355f2faa6 P 3e025d72db7c49159529617974c08232 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e025d72db7c49159529617974c08232" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 42039 } } peers { permanent_uuid: "01baffeacfa6457094831efd2c408c4e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 44633 } } peers { permanent_uuid: "468ecc83d41743dc981739ecb9b20067" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45535 } }
I20260429 14:04:36.495772 9585 catalog_manager.cc:5671] T f67edf65ccbd4e53967be2b355f2faa6 P 3e025d72db7c49159529617974c08232 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e025d72db7c49159529617974c08232 (127.2.197.195). New cstate: current_term: 1 leader_uuid: "3e025d72db7c49159529617974c08232" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e025d72db7c49159529617974c08232" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 42039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01baffeacfa6457094831efd2c408c4e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 44633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "468ecc83d41743dc981739ecb9b20067" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45535 } health_report { overall_health: UNKNOWN } } }
I20260429 14:04:36.516928 10885 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:36.514735 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48906 (local address 127.2.197.195:42039)
0429 14:04:36.514840 (+ 105us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:36.514842 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:36.514851 (+ 9us) server_negotiation.cc:408] Connection header received
0429 14:04:36.514887 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:36.514888 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:36.514902 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:36.514996 (+ 94us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:04:36.515257 (+ 261us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.515727 (+ 470us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:36.516385 (+ 658us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.516546 (+ 161us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:36.516576 (+ 30us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:04:36.516656 (+ 80us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:04:36.516743 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:36.516818 (+ 75us) server_negotiation.cc:300] Negotiation successful
0429 14:04:36.516843 (+ 25us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260429 14:04:36.518592 10673 raft_consensus.cc:1275] T f67edf65ccbd4e53967be2b355f2faa6 P 01baffeacfa6457094831efd2c408c4e [term 1 FOLLOWER]: Refusing update from remote peer 3e025d72db7c49159529617974c08232: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:04:36.518877 10537 raft_consensus.cc:1275] T f67edf65ccbd4e53967be2b355f2faa6 P 468ecc83d41743dc981739ecb9b20067 [term 1 FOLLOWER]: Refusing update from remote peer 3e025d72db7c49159529617974c08232: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:04:36.519186 10896 consensus_queue.cc:1048] T f67edf65ccbd4e53967be2b355f2faa6 P 3e025d72db7c49159529617974c08232 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01baffeacfa6457094831efd2c408c4e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 44633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:04:36.519335 10893 consensus_queue.cc:1048] T f67edf65ccbd4e53967be2b355f2faa6 P 3e025d72db7c49159529617974c08232 [LEADER]: Connected to new peer: Peer: permanent_uuid: "468ecc83d41743dc981739ecb9b20067" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:04:36.523905 10900 mvcc.cc:204] Tried to move back new op lower bound from 7280523167817588736 to 7280523167723139072. Current Snapshot: MvccSnapshot[applied={T|T < 7280523167817588736 or (T in {7280523167817588736})}]
I20260429 14:04:36.523831 10901 mvcc.cc:204] Tried to move back new op lower bound from 7280523167817588736 to 7280523167723139072. Current Snapshot: MvccSnapshot[applied={T|T < 7280523167817588736}]
I20260429 14:04:36.524277 9585 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:04:36.524359 9585 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:04:36.525740 9585 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f8a2e74b05bf42de93b558952af4f587: Sending DeleteTablet for 3 replicas of tablet f67edf65ccbd4e53967be2b355f2faa6
I20260429 14:04:36.526109 10517 tablet_service.cc:1558] Processing DeleteTablet for tablet f67edf65ccbd4e53967be2b355f2faa6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:04:36 UTC) from {username='oryx', principal='oryx/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:37764
I20260429 14:04:36.526115 10790 tablet_service.cc:1558] Processing DeleteTablet for tablet f67edf65ccbd4e53967be2b355f2faa6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:04:36 UTC) from {username='oryx', principal='oryx/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:48890
I20260429 14:04:36.526165 10653 tablet_service.cc:1558] Processing DeleteTablet for tablet f67edf65ccbd4e53967be2b355f2faa6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:04:36 UTC) from {username='oryx', principal='oryx/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:46518
I20260429 14:04:36.526453 10910 tablet_replica.cc:333] T f67edf65ccbd4e53967be2b355f2faa6 P 01baffeacfa6457094831efd2c408c4e: stopping tablet replica
I20260429 14:04:36.526546 10908 tablet_replica.cc:333] T f67edf65ccbd4e53967be2b355f2faa6 P 468ecc83d41743dc981739ecb9b20067: stopping tablet replica
I20260429 14:04:36.526553 10910 raft_consensus.cc:2243] T f67edf65ccbd4e53967be2b355f2faa6 P 01baffeacfa6457094831efd2c408c4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:04:36.526654 10910 raft_consensus.cc:2272] T f67edf65ccbd4e53967be2b355f2faa6 P 01baffeacfa6457094831efd2c408c4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:04:36.526671 10908 raft_consensus.cc:2243] T f67edf65ccbd4e53967be2b355f2faa6 P 468ecc83d41743dc981739ecb9b20067 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:04:36.527006 10908 raft_consensus.cc:2272] T f67edf65ccbd4e53967be2b355f2faa6 P 468ecc83d41743dc981739ecb9b20067 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:04:36.527530 10910 ts_tablet_manager.cc:1916] T f67edf65ccbd4e53967be2b355f2faa6 P 01baffeacfa6457094831efd2c408c4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:04:36.527766 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 9554
I20260429 14:04:36.528395 10908 ts_tablet_manager.cc:1916] T f67edf65ccbd4e53967be2b355f2faa6 P 468ecc83d41743dc981739ecb9b20067: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:04:36.530129 10910 ts_tablet_manager.cc:1929] T f67edf65ccbd4e53967be2b355f2faa6 P 01baffeacfa6457094831efd2c408c4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 14:04:36.530208 10910 log.cc:1199] T f67edf65ccbd4e53967be2b355f2faa6 P 01baffeacfa6457094831efd2c408c4e: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-1/wal/wals/f67edf65ccbd4e53967be2b355f2faa6
I20260429 14:04:36.530879 10910 ts_tablet_manager.cc:1950] T f67edf65ccbd4e53967be2b355f2faa6 P 01baffeacfa6457094831efd2c408c4e: Deleting consensus metadata
I20260429 14:04:36.534564 10909 tablet_replica.cc:333] T f67edf65ccbd4e53967be2b355f2faa6 P 3e025d72db7c49159529617974c08232: stopping tablet replica
I20260429 14:04:36.534780 10909 raft_consensus.cc:2243] T f67edf65ccbd4e53967be2b355f2faa6 P 3e025d72db7c49159529617974c08232 [term 1 LEADER]: Raft consensus shutting down.
W20260429 14:04:36.534931 10607 connection.cc:570] server connection from 127.0.0.1:46518 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260429 14:04:36.535334 10909 raft_consensus.cc:2272] T f67edf65ccbd4e53967be2b355f2faa6 P 3e025d72db7c49159529617974c08232 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:04:36.535934 10909 ts_tablet_manager.cc:1916] T f67edf65ccbd4e53967be2b355f2faa6 P 3e025d72db7c49159529617974c08232: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:04:36.536357 10908 ts_tablet_manager.cc:1929] T f67edf65ccbd4e53967be2b355f2faa6 P 468ecc83d41743dc981739ecb9b20067: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 14:04:36.536453 10908 log.cc:1199] T f67edf65ccbd4e53967be2b355f2faa6 P 468ecc83d41743dc981739ecb9b20067: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-0/wal/wals/f67edf65ccbd4e53967be2b355f2faa6
I20260429 14:04:36.536759 10908 ts_tablet_manager.cc:1950] T f67edf65ccbd4e53967be2b355f2faa6 P 468ecc83d41743dc981739ecb9b20067: Deleting consensus metadata
W20260429 14:04:36.537197 10470 connection.cc:463] server connection from 127.0.0.1:37764 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37764 (request call id 1) could send its response
I20260429 14:04:36.538152 10909 ts_tablet_manager.cc:1929] T f67edf65ccbd4e53967be2b355f2faa6 P 3e025d72db7c49159529617974c08232: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 14:04:36.538231 10909 log.cc:1199] T f67edf65ccbd4e53967be2b355f2faa6 P 3e025d72db7c49159529617974c08232: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-2/wal/wals/f67edf65ccbd4e53967be2b355f2faa6
I20260429 14:04:36.538538 10909 ts_tablet_manager.cc:1950] T f67edf65ccbd4e53967be2b355f2faa6 P 3e025d72db7c49159529617974c08232: Deleting consensus metadata
W20260429 14:04:36.538862 10746 connection.cc:463] server connection from 127.0.0.1:48890 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48890 (request call id 1) could send its response
I20260429 14:04:36.544914 10449 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:36.537676 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:47058 (local address 127.2.197.253:44465)
0429 14:04:36.537726 (+ 50us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:36.537728 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:36.537833 (+ 105us) server_negotiation.cc:408] Connection header received
0429 14:04:36.537926 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:36.537928 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:36.537940 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:36.537995 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:36.539059 (+ 1064us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.539533 (+ 474us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:36.540298 (+ 765us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.540485 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:36.541700 (+ 1215us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:36.541707 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:36.541707 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:36.541714 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:36.543775 (+ 2061us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:36.544168 (+ 393us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:36.544169 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:36.544170 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:36.544197 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:36.544360 (+ 163us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:36.544361 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:36.544362 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:36.544450 (+ 88us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:36.544550 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:36.544689 (+ 139us) server_negotiation.cc:300] Negotiation successful
0429 14:04:36.544769 (+ 80us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260429 14:04:36.545955 10871 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:36.538112 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36634 (local address 127.2.197.252:36013)
0429 14:04:36.538152 (+ 40us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:36.538155 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:36.538425 (+ 270us) server_negotiation.cc:408] Connection header received
0429 14:04:36.538551 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:36.538553 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:36.538570 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:36.538606 (+ 36us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:36.540252 (+ 1646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.541028 (+ 776us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:36.541829 (+ 801us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:36.542011 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:36.543070 (+ 1059us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:36.543073 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:36.543074 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:36.543081 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:36.544811 (+ 1730us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:36.545163 (+ 352us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:36.545165 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:36.545165 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:36.545192 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:36.545366 (+ 174us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:36.545367 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:36.545367 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:36.545472 (+ 105us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:36.545558 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:36.545755 (+ 197us) server_negotiation.cc:300] Negotiation successful
0429 14:04:36.545839 (+ 84us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260429 14:04:37.523658 10917 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:37.523345 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.2.197.193:33969)
0429 14:04:37.523523 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:37.523596 (+ 73us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":100,"threads_started":1}
I20260429 14:04:37.523841 10918 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:37.523516 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.2.197.194:51739)
0429 14:04:37.523672 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:37.523716 (+ 44us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":57,"threads_started":1}
W20260429 14:04:37.524326 10720 heartbeater.cc:646] Failed to heartbeat to 127.2.197.254:40335 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
W20260429 14:04:37.524335 10584 heartbeater.cc:646] Failed to heartbeat to 127.2.197.254:40335 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
I20260429 14:04:37.524374 10919 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:37.524098 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.2.197.195:36383)
0429 14:04:37.524248 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:37.524322 (+ 74us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":47,"threads_started":1}
W20260429 14:04:37.524761 10857 heartbeater.cc:646] Failed to heartbeat to 127.2.197.254:40335 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
I20260429 14:04:38.027573 10920 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8a2e74b05bf42de93b558952af4f587)
I20260429 14:04:38.027709 10920 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: false } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: false } }
I20260429 14:04:38.027875 10921 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8a2e74b05bf42de93b558952af4f587)
I20260429 14:04:38.028005 10921 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: false } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: false } }
I20260429 14:04:38.028097 10920 leader_election.cc:290] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8a2e74b05bf42de93b558952af4f587 (127.2.197.254:40335), 7de05a456e054b9f83790eb862246127 (127.2.197.253:44465)
I20260429 14:04:38.028275 10921 leader_election.cc:290] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8a2e74b05bf42de93b558952af4f587 (127.2.197.254:40335), 9782bdb83d3143c6a2f310e1c7de398f (127.2.197.252:36013)
I20260429 14:04:38.028455 10207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9782bdb83d3143c6a2f310e1c7de398f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7de05a456e054b9f83790eb862246127" is_pre_election: true
I20260429 14:04:38.028479 10922 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:38.028237 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.0.0.1:59696)
0429 14:04:38.028393 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:38.028437 (+ 44us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":58,"threads_started":1}
W20260429 14:04:38.028575 10376 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
I20260429 14:04:38.028573 10207 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9782bdb83d3143c6a2f310e1c7de398f in term 1.
I20260429 14:04:38.028474 10400 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7de05a456e054b9f83790eb862246127" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9782bdb83d3143c6a2f310e1c7de398f" is_pre_election: true
I20260429 14:04:38.028661 10400 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7de05a456e054b9f83790eb862246127 in term 1.
I20260429 14:04:38.028729 10376 leader_election.cc:304] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7de05a456e054b9f83790eb862246127, 9782bdb83d3143c6a2f310e1c7de398f; no voters:
I20260429 14:04:38.028836 10920 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260429 14:04:38.028880 10920 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [term 1 FOLLOWER]: Starting leader election (detected failure of leader f8a2e74b05bf42de93b558952af4f587)
I20260429 14:04:38.028841 10180 leader_election.cc:304] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7de05a456e054b9f83790eb862246127, 9782bdb83d3143c6a2f310e1c7de398f; no voters:
I20260429 14:04:38.028910 10920 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [term 1 FOLLOWER]: Advancing to term 2
I20260429 14:04:38.028956 10921 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260429 14:04:38.028852 10923 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:38.028422 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.0.0.1:59708)
0429 14:04:38.028516 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:38.028747 (+ 231us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":44,"threads_started":1}
I20260429 14:04:38.029042 10922 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:38.028899 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.0.0.1:59714)
0429 14:04:38.028987 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:38.029010 (+ 23us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":52,"threads_started":1}
I20260429 14:04:38.029042 10921 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f8a2e74b05bf42de93b558952af4f587)
W20260429 14:04:38.029158 10181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
W20260429 14:04:38.029157 10376 leader_election.cc:336] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f8a2e74b05bf42de93b558952af4f587 (127.2.197.254:40335): Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
I20260429 14:04:38.029179 10921 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [term 1 FOLLOWER]: Advancing to term 2
I20260429 14:04:38.029524 10923 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:38.029440 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.0.0.1:59724)
0429 14:04:38.029470 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:38.029495 (+ 25us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260429 14:04:38.029623 10181 leader_election.cc:336] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f8a2e74b05bf42de93b558952af4f587 (127.2.197.254:40335): Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
I20260429 14:04:38.029873 10920 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: false } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: false } }
I20260429 14:04:38.030021 10920 leader_election.cc:290] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [CANDIDATE]: Term 2 election: Requested vote from peers f8a2e74b05bf42de93b558952af4f587 (127.2.197.254:40335), 7de05a456e054b9f83790eb862246127 (127.2.197.253:44465)
I20260429 14:04:38.030027 10921 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: false } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: false } }
I20260429 14:04:38.030177 10921 leader_election.cc:290] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [CANDIDATE]: Term 2 election: Requested vote from peers f8a2e74b05bf42de93b558952af4f587 (127.2.197.254:40335), 9782bdb83d3143c6a2f310e1c7de398f (127.2.197.252:36013)
I20260429 14:04:38.030340 10925 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:38.030153 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.0.0.1:59736)
0429 14:04:38.030199 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:38.030307 (+ 108us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260429 14:04:38.030346 10923 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:38.030248 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.0.0.1:59746)
0429 14:04:38.030299 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:38.030324 (+ 25us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260429 14:04:38.030404 10400 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7de05a456e054b9f83790eb862246127" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9782bdb83d3143c6a2f310e1c7de398f"
I20260429 14:04:38.030357 10207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9782bdb83d3143c6a2f310e1c7de398f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7de05a456e054b9f83790eb862246127"
I20260429 14:04:38.030547 10207 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9782bdb83d3143c6a2f310e1c7de398f in current term 2: Already voted for candidate 7de05a456e054b9f83790eb862246127 in this term.
I20260429 14:04:38.030560 10400 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7de05a456e054b9f83790eb862246127 in current term 2: Already voted for candidate 9782bdb83d3143c6a2f310e1c7de398f in this term.
I20260429 14:04:38.030897 10923 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:38.030786 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.0.0.1:59750)
0429 14:04:38.030842 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:38.030878 (+ 36us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260429 14:04:38.030983 10181 leader_election.cc:336] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f8a2e74b05bf42de93b558952af4f587 (127.2.197.254:40335): Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
I20260429 14:04:38.031024 10181 leader_election.cc:304] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7de05a456e054b9f83790eb862246127; no voters: 9782bdb83d3143c6a2f310e1c7de398f, f8a2e74b05bf42de93b558952af4f587
I20260429 14:04:38.031068 10925 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:38.030922 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.0.0.1:59762)
0429 14:04:38.031018 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:38.031042 (+ 24us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260429 14:04:38.031131 10921 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260429 14:04:38.031162 10376 leader_election.cc:336] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f8a2e74b05bf42de93b558952af4f587 (127.2.197.254:40335): Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
I20260429 14:04:38.031204 10376 leader_election.cc:304] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9782bdb83d3143c6a2f310e1c7de398f; no voters: 7de05a456e054b9f83790eb862246127, f8a2e74b05bf42de93b558952af4f587
I20260429 14:04:38.031283 10920 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260429 14:04:38.525014 10927 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:38.524774 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.2.197.193:38405)
0429 14:04:38.524916 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:38.524958 (+ 42us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":76,"threads_started":1}
I20260429 14:04:38.525394 10929 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:38.525180 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.2.197.195:58903)
0429 14:04:38.525314 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:38.525349 (+ 35us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":65,"threads_started":1}
I20260429 14:04:38.525417 10928 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:38.524945 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.2.197.194:47785)
0429 14:04:38.525340 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:38.525373 (+ 33us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":53,"threads_started":1}
I20260429 14:04:39.526211 10931 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:39.525976 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.2.197.193:52365)
0429 14:04:39.526108 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:39.526147 (+ 39us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":66,"threads_started":1}
I20260429 14:04:39.526255 10932 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:39.526017 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.2.197.194:56781)
0429 14:04:39.526128 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:39.526205 (+ 77us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":55,"threads_started":1}
I20260429 14:04:39.526501 10930 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:39.525977 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.2.197.195:44373)
0429 14:04:39.526423 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:39.526458 (+ 35us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":66,"threads_started":1}
I20260429 14:04:39.720116 10933 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:04:39.720223 10933 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: false } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: false } }
I20260429 14:04:39.720356 10933 leader_election.cc:290] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8a2e74b05bf42de93b558952af4f587 (127.2.197.254:40335), 7de05a456e054b9f83790eb862246127 (127.2.197.253:44465)
I20260429 14:04:39.720738 10934 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:39.720551 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.0.0.1:57278)
0429 14:04:39.720651 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:39.720690 (+ 39us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":49,"threads_started":1}
I20260429 14:04:39.720839 10207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9782bdb83d3143c6a2f310e1c7de398f" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7de05a456e054b9f83790eb862246127" is_pre_election: true
I20260429 14:04:39.720935 10207 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9782bdb83d3143c6a2f310e1c7de398f in term 2.
I20260429 14:04:39.721220 10376 leader_election.cc:304] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7de05a456e054b9f83790eb862246127, 9782bdb83d3143c6a2f310e1c7de398f; no voters:
I20260429 14:04:39.721273 10934 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:39.721152 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.0.0.1:57290)
0429 14:04:39.721202 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:39.721229 (+ 27us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260429 14:04:39.721310 10376 leader_election.cc:336] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f8a2e74b05bf42de93b558952af4f587 (127.2.197.254:40335): Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
I20260429 14:04:39.721342 10933 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260429 14:04:39.721419 10933 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 14:04:39.721455 10933 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [term 2 FOLLOWER]: Advancing to term 3
I20260429 14:04:39.722190 10933 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: false } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: false } }
I20260429 14:04:39.722363 10933 leader_election.cc:290] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [CANDIDATE]: Term 3 election: Requested vote from peers f8a2e74b05bf42de93b558952af4f587 (127.2.197.254:40335), 7de05a456e054b9f83790eb862246127 (127.2.197.253:44465)
I20260429 14:04:39.722549 10934 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:39.722467 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.0.0.1:57292)
0429 14:04:39.722499 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:39.722525 (+ 26us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260429 14:04:39.722746 10207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9782bdb83d3143c6a2f310e1c7de398f" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7de05a456e054b9f83790eb862246127"
I20260429 14:04:39.722878 10207 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [term 2 FOLLOWER]: Advancing to term 3
I20260429 14:04:39.722900 10934 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:39.722811 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.0.0.1:57304)
0429 14:04:39.722853 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:39.722875 (+ 22us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260429 14:04:39.723026 10376 leader_election.cc:336] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer f8a2e74b05bf42de93b558952af4f587 (127.2.197.254:40335): Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
I20260429 14:04:39.723649 10207 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9782bdb83d3143c6a2f310e1c7de398f in term 3.
I20260429 14:04:39.723906 10376 leader_election.cc:304] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7de05a456e054b9f83790eb862246127, 9782bdb83d3143c6a2f310e1c7de398f; no voters: f8a2e74b05bf42de93b558952af4f587
I20260429 14:04:39.724036 10933 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [term 3 FOLLOWER]: Leader election won for term 3
I20260429 14:04:39.724177 10933 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [term 3 LEADER]: Becoming Leader. State: Replica: 9782bdb83d3143c6a2f310e1c7de398f, State: Running, Role: LEADER
I20260429 14:04:39.724264 10933 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: false } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: false } }
I20260429 14:04:39.724797 10936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9782bdb83d3143c6a2f310e1c7de398f. Latest consensus state: current_term: 3 leader_uuid: "9782bdb83d3143c6a2f310e1c7de398f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: false } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: false } } }
I20260429 14:04:39.724900 10936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [sys.catalog]: This master's current role is: LEADER
I20260429 14:04:39.725103 10938 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:04:39.725494 10938 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:04:39.725569 10938 catalog_manager.cc:1269] Loaded cluster ID: bfb901df49664bdeb1e28c49a5039180
I20260429 14:04:39.725586 10938 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:04:39.725826 10938 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:04:39.725919 10938 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f: Loaded TSK: 0
I20260429 14:04:39.726110 10938 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 14:04:39.782986 10389 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36634:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 14:04:39.784329 10389 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 14:04:39.785454 10934 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:39.785306 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.0.0.1:57324)
0429 14:04:39.785381 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:39.785420 (+ 39us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260429 14:04:39.785495 10207 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [term 3 FOLLOWER]: Refusing update from remote peer 9782bdb83d3143c6a2f310e1c7de398f: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260429 14:04:39.785929 10934 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:39.785802 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.0.0.1:57334)
0429 14:04:39.785874 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:39.785909 (+ 35us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260429 14:04:39.785890 10933 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260429 14:04:39.786024 10376 consensus_peers.cc:597] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f -> Peer f8a2e74b05bf42de93b558952af4f587 (127.2.197.254:40335): Couldn't send request to peer f8a2e74b05bf42de93b558952af4f587. Status: Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260429 14:04:39.786736 10940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9782bdb83d3143c6a2f310e1c7de398f. Latest consensus state: current_term: 3 leader_uuid: "9782bdb83d3143c6a2f310e1c7de398f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: false } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: false } } }
I20260429 14:04:39.786914 10940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [sys.catalog]: This master's current role is: FOLLOWER
I20260429 14:04:39.787361 10933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "9782bdb83d3143c6a2f310e1c7de398f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: false } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: false } } }
I20260429 14:04:39.787434 10940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "9782bdb83d3143c6a2f310e1c7de398f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f8a2e74b05bf42de93b558952af4f587" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40335 } } peers { permanent_uuid: "7de05a456e054b9f83790eb862246127" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44465 } attrs { promote: false } } peers { permanent_uuid: "9782bdb83d3143c6a2f310e1c7de398f" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 36013 } attrs { promote: false } } }
I20260429 14:04:39.787475 10941 mvcc.cc:204] Tried to move back new op lower bound from 7280523181198864384 to 7280523180951883776. Current Snapshot: MvccSnapshot[applied={T|T < 7280523181198864384}]
I20260429 14:04:39.787678 10933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f [sys.catalog]: This master's current role is: LEADER
I20260429 14:04:39.787515 10940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7de05a456e054b9f83790eb862246127 [sys.catalog]: This master's current role is: FOLLOWER
I20260429 14:04:39.792109 10934 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:39.789543 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:42039 (local address 127.0.0.1:59428)
0429 14:04:39.789780 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:39.789789 (+ 9us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:39.789838 (+ 49us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:39.790084 (+ 246us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:39.790085 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:39.790086 (+ 1us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:04:39.790432 (+ 346us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:39.790435 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:39.791148 (+ 713us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:39.791148 (+ 0us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:39.791835 (+ 687us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:39.791838 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:39.791913 (+ 75us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:39.791918 (+ 5us) client_negotiation.cc:770] Sending connection context
0429 14:04:39.791956 (+ 38us) client_negotiation.cc:241] Negotiation successful
0429 14:04:39.792027 (+ 71us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":142,"threads_started":1}
I20260429 14:04:39.792676 10949 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:39.789446 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:44633 (local address 127.0.0.1:52514)
0429 14:04:39.789724 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:39.789742 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:39.789814 (+ 72us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:39.790034 (+ 220us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:39.790035 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:39.790038 (+ 3us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:04:39.790204 (+ 166us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:39.790206 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:39.791316 (+ 1110us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:39.791317 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:39.792508 (+ 1191us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:39.792510 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:39.792555 (+ 45us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:39.792559 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 14:04:39.792581 (+ 22us) client_negotiation.cc:241] Negotiation successful
0429 14:04:39.792615 (+ 34us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256}
I20260429 14:04:39.792878 10950 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:39.789660 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:59428 (local address 127.2.197.195:42039)
0429 14:04:39.789761 (+ 101us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:39.789763 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:39.789956 (+ 193us) server_negotiation.cc:408] Connection header received
0429 14:04:39.789986 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:39.789987 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:39.790000 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:39.790081 (+ 81us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:04:39.790527 (+ 446us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:39.791050 (+ 523us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:39.792301 (+ 1251us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:39.792714 (+ 413us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:39.792745 (+ 31us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:39.792770 (+ 25us) server_negotiation.cc:300] Negotiation successful
0429 14:04:39.792813 (+ 43us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":55,"threads_started":1}
I20260429 14:04:39.793092 10790 tablet_service.cc:1511] Processing CreateTablet for tablet b918b04d3c334e4aa7dad1fc77e29c06 (DEFAULT_TABLE table=test-table [id=1d5b0b6a75ea4cbdb16870522b9c812f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:04:39.793234 10790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b918b04d3c334e4aa7dad1fc77e29c06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:39.793531 10948 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:39.789558 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:52514 (local address 127.2.197.194:44633)
0429 14:04:39.789810 (+ 252us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:39.789812 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:39.789847 (+ 35us) server_negotiation.cc:408] Connection header received
0429 14:04:39.789877 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:39.789878 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:39.789890 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:39.789948 (+ 58us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:04:39.790325 (+ 377us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:39.791221 (+ 896us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:39.792922 (+ 1701us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:39.793402 (+ 480us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:39.793411 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:39.793435 (+ 24us) server_negotiation.cc:300] Negotiation successful
0429 14:04:39.793471 (+ 36us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":69,"threads_started":1}
I20260429 14:04:39.793824 10653 tablet_service.cc:1511] Processing CreateTablet for tablet b918b04d3c334e4aa7dad1fc77e29c06 (DEFAULT_TABLE table=test-table [id=1d5b0b6a75ea4cbdb16870522b9c812f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:04:39.793932 10653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b918b04d3c334e4aa7dad1fc77e29c06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:39.795042 10947 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:39.789540 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:45576 (local address 127.2.197.193:45535)
0429 14:04:39.790350 (+ 810us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:39.790352 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:39.790369 (+ 17us) server_negotiation.cc:408] Connection header received
0429 14:04:39.790516 (+ 147us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:39.790517 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:39.790528 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:39.790586 (+ 58us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:04:39.790905 (+ 319us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:39.791739 (+ 834us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:39.794376 (+ 2637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:39.794882 (+ 506us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:39.794890 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:39.794919 (+ 29us) server_negotiation.cc:300] Negotiation successful
0429 14:04:39.794969 (+ 50us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":788,"thread_start_us":65,"threads_started":1}
I20260429 14:04:39.793002 10946 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:39.789460 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:45535 (local address 127.0.0.1:45576)
0429 14:04:39.789736 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:39.789817 (+ 81us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:39.789874 (+ 57us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:39.790636 (+ 762us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:39.790637 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:39.790639 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:04:39.790815 (+ 176us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:39.790816 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:39.791939 (+ 1123us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:39.791940 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:39.792813 (+ 873us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:39.792816 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:39.792877 (+ 61us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:39.792880 (+ 3us) client_negotiation.cc:770] Sending connection context
0429 14:04:39.792915 (+ 35us) client_negotiation.cc:241] Negotiation successful
0429 14:04:39.792948 (+ 33us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":86,"threads_started":1}
I20260429 14:04:39.795204 10951 tablet_bootstrap.cc:492] T b918b04d3c334e4aa7dad1fc77e29c06 P 3e025d72db7c49159529617974c08232: Bootstrap starting.
I20260429 14:04:39.795567 10517 tablet_service.cc:1511] Processing CreateTablet for tablet b918b04d3c334e4aa7dad1fc77e29c06 (DEFAULT_TABLE table=test-table [id=1d5b0b6a75ea4cbdb16870522b9c812f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:04:39.795653 10951 tablet_bootstrap.cc:654] T b918b04d3c334e4aa7dad1fc77e29c06 P 3e025d72db7c49159529617974c08232: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:39.795684 10517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b918b04d3c334e4aa7dad1fc77e29c06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:39.796306 10951 tablet_bootstrap.cc:492] T b918b04d3c334e4aa7dad1fc77e29c06 P 3e025d72db7c49159529617974c08232: No bootstrap required, opened a new log
I20260429 14:04:39.796352 10951 ts_tablet_manager.cc:1403] T b918b04d3c334e4aa7dad1fc77e29c06 P 3e025d72db7c49159529617974c08232: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:39.796458 10951 raft_consensus.cc:359] T b918b04d3c334e4aa7dad1fc77e29c06 P 3e025d72db7c49159529617974c08232 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01baffeacfa6457094831efd2c408c4e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 44633 } } peers { permanent_uuid: "3e025d72db7c49159529617974c08232" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 42039 } } peers { permanent_uuid: "468ecc83d41743dc981739ecb9b20067" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45535 } }
I20260429 14:04:39.796554 10951 raft_consensus.cc:385] T b918b04d3c334e4aa7dad1fc77e29c06 P 3e025d72db7c49159529617974c08232 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:39.796569 10951 raft_consensus.cc:740] T b918b04d3c334e4aa7dad1fc77e29c06 P 3e025d72db7c49159529617974c08232 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e025d72db7c49159529617974c08232, State: Initialized, Role: FOLLOWER
I20260429 14:04:39.796744 10951 consensus_queue.cc:260] T b918b04d3c334e4aa7dad1fc77e29c06 P 3e025d72db7c49159529617974c08232 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01baffeacfa6457094831efd2c408c4e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 44633 } } peers { permanent_uuid: "3e025d72db7c49159529617974c08232" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 42039 } } peers { permanent_uuid: "468ecc83d41743dc981739ecb9b20067" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45535 } }
I20260429 14:04:39.796926 10952 tablet_bootstrap.cc:492] T b918b04d3c334e4aa7dad1fc77e29c06 P 01baffeacfa6457094831efd2c408c4e: Bootstrap starting.
I20260429 14:04:39.796979 10951 ts_tablet_manager.cc:1434] T b918b04d3c334e4aa7dad1fc77e29c06 P 3e025d72db7c49159529617974c08232: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:39.796799 10954 tablet_bootstrap.cc:492] T b918b04d3c334e4aa7dad1fc77e29c06 P 468ecc83d41743dc981739ecb9b20067: Bootstrap starting.
I20260429 14:04:39.797363 10952 tablet_bootstrap.cc:654] T b918b04d3c334e4aa7dad1fc77e29c06 P 01baffeacfa6457094831efd2c408c4e: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:39.797381 10954 tablet_bootstrap.cc:654] T b918b04d3c334e4aa7dad1fc77e29c06 P 468ecc83d41743dc981739ecb9b20067: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:39.797962 10856 heartbeater.cc:499] Master 127.2.197.252:36013 was elected leader, sending a full tablet report...
I20260429 14:04:39.798115 10930 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:39.797968 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.2.197.195:49699)
0429 14:04:39.798047 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:39.798078 (+ 31us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260429 14:04:39.798144 10952 tablet_bootstrap.cc:492] T b918b04d3c334e4aa7dad1fc77e29c06 P 01baffeacfa6457094831efd2c408c4e: No bootstrap required, opened a new log
I20260429 14:04:39.798187 10952 ts_tablet_manager.cc:1403] T b918b04d3c334e4aa7dad1fc77e29c06 P 01baffeacfa6457094831efd2c408c4e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:39.798357 10952 raft_consensus.cc:359] T b918b04d3c334e4aa7dad1fc77e29c06 P 01baffeacfa6457094831efd2c408c4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01baffeacfa6457094831efd2c408c4e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 44633 } } peers { permanent_uuid: "3e025d72db7c49159529617974c08232" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 42039 } } peers { permanent_uuid: "468ecc83d41743dc981739ecb9b20067" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45535 } }
I20260429 14:04:39.798436 10952 raft_consensus.cc:385] T b918b04d3c334e4aa7dad1fc77e29c06 P 01baffeacfa6457094831efd2c408c4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:39.798425 10930 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:39.798340 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.2.197.195:36299)
0429 14:04:39.798372 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:39.798397 (+ 25us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260429 14:04:39.798461 10952 raft_consensus.cc:740] T b918b04d3c334e4aa7dad1fc77e29c06 P 01baffeacfa6457094831efd2c408c4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01baffeacfa6457094831efd2c408c4e, State: Initialized, Role: FOLLOWER
I20260429 14:04:39.798632 10952 consensus_queue.cc:260] T b918b04d3c334e4aa7dad1fc77e29c06 P 01baffeacfa6457094831efd2c408c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01baffeacfa6457094831efd2c408c4e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 44633 } } peers { permanent_uuid: "3e025d72db7c49159529617974c08232" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 42039 } } peers { permanent_uuid: "468ecc83d41743dc981739ecb9b20067" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45535 } }
I20260429 14:04:39.798866 10952 ts_tablet_manager.cc:1434] T b918b04d3c334e4aa7dad1fc77e29c06 P 01baffeacfa6457094831efd2c408c4e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:39.799165 10932 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:39.799069 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.2.197.194:45521)
0429 14:04:39.799114 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:39.799137 (+ 23us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260429 14:04:39.799214 10954 tablet_bootstrap.cc:492] T b918b04d3c334e4aa7dad1fc77e29c06 P 468ecc83d41743dc981739ecb9b20067: No bootstrap required, opened a new log
I20260429 14:04:39.799271 10954 ts_tablet_manager.cc:1403] T b918b04d3c334e4aa7dad1fc77e29c06 P 468ecc83d41743dc981739ecb9b20067: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260429 14:04:39.799409 10954 raft_consensus.cc:359] T b918b04d3c334e4aa7dad1fc77e29c06 P 468ecc83d41743dc981739ecb9b20067 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01baffeacfa6457094831efd2c408c4e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 44633 } } peers { permanent_uuid: "3e025d72db7c49159529617974c08232" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 42039 } } peers { permanent_uuid: "468ecc83d41743dc981739ecb9b20067" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45535 } }
I20260429 14:04:39.799487 10954 raft_consensus.cc:385] T b918b04d3c334e4aa7dad1fc77e29c06 P 468ecc83d41743dc981739ecb9b20067 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:39.799508 10954 raft_consensus.cc:740] T b918b04d3c334e4aa7dad1fc77e29c06 P 468ecc83d41743dc981739ecb9b20067 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 468ecc83d41743dc981739ecb9b20067, State: Initialized, Role: FOLLOWER
I20260429 14:04:39.799680 10954 consensus_queue.cc:260] T b918b04d3c334e4aa7dad1fc77e29c06 P 468ecc83d41743dc981739ecb9b20067 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01baffeacfa6457094831efd2c408c4e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 44633 } } peers { permanent_uuid: "3e025d72db7c49159529617974c08232" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 42039 } } peers { permanent_uuid: "468ecc83d41743dc981739ecb9b20067" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45535 } }
I20260429 14:04:39.799916 10954 ts_tablet_manager.cc:1434] T b918b04d3c334e4aa7dad1fc77e29c06 P 468ecc83d41743dc981739ecb9b20067: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:39.800171 10931 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:39.800075 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.2.197.193:56909)
0429 14:04:39.800119 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:39.800142 (+ 23us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260429 14:04:39.800230 10719 heartbeater.cc:499] Master 127.2.197.252:36013 was elected leader, sending a full tablet report...
I20260429 14:04:39.800335 10583 heartbeater.cc:499] Master 127.2.197.252:36013 was elected leader, sending a full tablet report...
I20260429 14:04:39.800437 10931 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:39.800358 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.2.197.193:38709)
0429 14:04:39.800389 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:39.800411 (+ 22us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260429 14:04:39.800957 10932 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:39.800820 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.2.197.194:60919)
0429 14:04:39.800876 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:39.800930 (+ 54us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260429 14:04:39.837450 10586 tablet.cc:2404] T b918b04d3c334e4aa7dad1fc77e29c06 P 468ecc83d41743dc981739ecb9b20067: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260429 14:04:39.844070 10859 tablet.cc:2404] T b918b04d3c334e4aa7dad1fc77e29c06 P 3e025d72db7c49159529617974c08232: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 14:04:40.017429 10955 raft_consensus.cc:493] T b918b04d3c334e4aa7dad1fc77e29c06 P 3e025d72db7c49159529617974c08232 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:04:40.017588 10955 raft_consensus.cc:515] T b918b04d3c334e4aa7dad1fc77e29c06 P 3e025d72db7c49159529617974c08232 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01baffeacfa6457094831efd2c408c4e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 44633 } } peers { permanent_uuid: "3e025d72db7c49159529617974c08232" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 42039 } } peers { permanent_uuid: "468ecc83d41743dc981739ecb9b20067" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45535 } }
I20260429 14:04:40.017820 10955 leader_election.cc:290] T b918b04d3c334e4aa7dad1fc77e29c06 P 3e025d72db7c49159529617974c08232 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01baffeacfa6457094831efd2c408c4e (127.2.197.194:44633), 468ecc83d41743dc981739ecb9b20067 (127.2.197.193:45535)
I20260429 14:04:40.017979 10673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b918b04d3c334e4aa7dad1fc77e29c06" candidate_uuid: "3e025d72db7c49159529617974c08232" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01baffeacfa6457094831efd2c408c4e" is_pre_election: true
I20260429 14:04:40.017997 10537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b918b04d3c334e4aa7dad1fc77e29c06" candidate_uuid: "3e025d72db7c49159529617974c08232" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "468ecc83d41743dc981739ecb9b20067" is_pre_election: true
I20260429 14:04:40.018064 10673 raft_consensus.cc:2468] T b918b04d3c334e4aa7dad1fc77e29c06 P 01baffeacfa6457094831efd2c408c4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e025d72db7c49159529617974c08232 in term 0.
I20260429 14:04:40.018065 10537 raft_consensus.cc:2468] T b918b04d3c334e4aa7dad1fc77e29c06 P 468ecc83d41743dc981739ecb9b20067 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e025d72db7c49159529617974c08232 in term 0.
I20260429 14:04:40.018347 10744 leader_election.cc:304] T b918b04d3c334e4aa7dad1fc77e29c06 P 3e025d72db7c49159529617974c08232 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01baffeacfa6457094831efd2c408c4e, 3e025d72db7c49159529617974c08232; no voters:
I20260429 14:04:40.018594 10955 raft_consensus.cc:2804] T b918b04d3c334e4aa7dad1fc77e29c06 P 3e025d72db7c49159529617974c08232 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 14:04:40.018797 10955 raft_consensus.cc:493] T b918b04d3c334e4aa7dad1fc77e29c06 P 3e025d72db7c49159529617974c08232 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 14:04:40.018862 10955 raft_consensus.cc:3060] T b918b04d3c334e4aa7dad1fc77e29c06 P 3e025d72db7c49159529617974c08232 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:40.019585 10955 raft_consensus.cc:515] T b918b04d3c334e4aa7dad1fc77e29c06 P 3e025d72db7c49159529617974c08232 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01baffeacfa6457094831efd2c408c4e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 44633 } } peers { permanent_uuid: "3e025d72db7c49159529617974c08232" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 42039 } } peers { permanent_uuid: "468ecc83d41743dc981739ecb9b20067" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45535 } }
I20260429 14:04:40.019765 10955 leader_election.cc:290] T b918b04d3c334e4aa7dad1fc77e29c06 P 3e025d72db7c49159529617974c08232 [CANDIDATE]: Term 1 election: Requested vote from peers 01baffeacfa6457094831efd2c408c4e (127.2.197.194:44633), 468ecc83d41743dc981739ecb9b20067 (127.2.197.193:45535)
I20260429 14:04:40.020000 10537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b918b04d3c334e4aa7dad1fc77e29c06" candidate_uuid: "3e025d72db7c49159529617974c08232" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "468ecc83d41743dc981739ecb9b20067"
I20260429 14:04:40.020115 10537 raft_consensus.cc:3060] T b918b04d3c334e4aa7dad1fc77e29c06 P 468ecc83d41743dc981739ecb9b20067 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:40.020085 10673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b918b04d3c334e4aa7dad1fc77e29c06" candidate_uuid: "3e025d72db7c49159529617974c08232" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01baffeacfa6457094831efd2c408c4e"
I20260429 14:04:40.020154 10673 raft_consensus.cc:3060] T b918b04d3c334e4aa7dad1fc77e29c06 P 01baffeacfa6457094831efd2c408c4e [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:40.020896 10673 raft_consensus.cc:2468] T b918b04d3c334e4aa7dad1fc77e29c06 P 01baffeacfa6457094831efd2c408c4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e025d72db7c49159529617974c08232 in term 1.
I20260429 14:04:40.020891 10537 raft_consensus.cc:2468] T b918b04d3c334e4aa7dad1fc77e29c06 P 468ecc83d41743dc981739ecb9b20067 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e025d72db7c49159529617974c08232 in term 1.
I20260429 14:04:40.021132 10744 leader_election.cc:304] T b918b04d3c334e4aa7dad1fc77e29c06 P 3e025d72db7c49159529617974c08232 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01baffeacfa6457094831efd2c408c4e, 3e025d72db7c49159529617974c08232; no voters:
I20260429 14:04:40.021237 10955 raft_consensus.cc:2804] T b918b04d3c334e4aa7dad1fc77e29c06 P 3e025d72db7c49159529617974c08232 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:04:40.021412 10955 raft_consensus.cc:697] T b918b04d3c334e4aa7dad1fc77e29c06 P 3e025d72db7c49159529617974c08232 [term 1 LEADER]: Becoming Leader. State: Replica: 3e025d72db7c49159529617974c08232, State: Running, Role: LEADER
I20260429 14:04:40.021538 10955 consensus_queue.cc:237] T b918b04d3c334e4aa7dad1fc77e29c06 P 3e025d72db7c49159529617974c08232 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01baffeacfa6457094831efd2c408c4e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 44633 } } peers { permanent_uuid: "3e025d72db7c49159529617974c08232" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 42039 } } peers { permanent_uuid: "468ecc83d41743dc981739ecb9b20067" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45535 } }
I20260429 14:04:40.021772 10930 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:40.021657 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.2.197.195:36091)
0429 14:04:40.021703 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:40.021733 (+ 30us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260429 14:04:40.022320 10389 catalog_manager.cc:5671] T b918b04d3c334e4aa7dad1fc77e29c06 P 3e025d72db7c49159529617974c08232 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e025d72db7c49159529617974c08232 (127.2.197.195). New cstate: current_term: 1 leader_uuid: "3e025d72db7c49159529617974c08232" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01baffeacfa6457094831efd2c408c4e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 44633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e025d72db7c49159529617974c08232" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 42039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "468ecc83d41743dc981739ecb9b20067" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45535 } health_report { overall_health: UNKNOWN } } }
I20260429 14:04:40.080323 10950 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:40.077441 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:59430 (local address 127.2.197.195:42039)
0429 14:04:40.077647 (+ 206us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:40.077649 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:40.077660 (+ 11us) server_negotiation.cc:408] Connection header received
0429 14:04:40.077685 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:40.077686 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:40.077697 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:40.077826 (+ 129us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:04:40.078141 (+ 315us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:40.078979 (+ 838us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:40.079680 (+ 701us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:40.079853 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:40.079875 (+ 22us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:04:40.079903 (+ 28us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:04:40.079957 (+ 54us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:40.080134 (+ 177us) server_negotiation.cc:300] Negotiation successful
0429 14:04:40.080213 (+ 79us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175}
I20260429 14:04:40.081509 10673 raft_consensus.cc:1275] T b918b04d3c334e4aa7dad1fc77e29c06 P 01baffeacfa6457094831efd2c408c4e [term 1 FOLLOWER]: Refusing update from remote peer 3e025d72db7c49159529617974c08232: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:04:40.081617 10537 raft_consensus.cc:1275] T b918b04d3c334e4aa7dad1fc77e29c06 P 468ecc83d41743dc981739ecb9b20067 [term 1 FOLLOWER]: Refusing update from remote peer 3e025d72db7c49159529617974c08232: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:04:40.081815 10960 consensus_queue.cc:1048] T b918b04d3c334e4aa7dad1fc77e29c06 P 3e025d72db7c49159529617974c08232 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01baffeacfa6457094831efd2c408c4e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 44633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:04:40.081926 10955 consensus_queue.cc:1048] T b918b04d3c334e4aa7dad1fc77e29c06 P 3e025d72db7c49159529617974c08232 [LEADER]: Connected to new peer: Peer: permanent_uuid: "468ecc83d41743dc981739ecb9b20067" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:04:40.082727 10932 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:40.082615 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.2.197.194:54897)
0429 14:04:40.082661 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:40.082694 (+ 33us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260429 14:04:40.082875 10931 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:40.082740 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.2.197.193:40711)
0429 14:04:40.082810 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:40.082846 (+ 36us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260429 14:04:40.084220 10931 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:40.084060 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.2.197.193:40843)
0429 14:04:40.084168 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:40.084194 (+ 26us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260429 14:04:40.085369 10932 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:40.085257 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.2.197.194:37445)
0429 14:04:40.085301 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:40.085330 (+ 29us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260429 14:04:40.085644 10963 mvcc.cc:204] Tried to move back new op lower bound from 7280523182411636736 to 7280523182168858624. Current Snapshot: MvccSnapshot[applied={T|T < 7280523182411636736 or (T in {7280523182411636736})}]
I20260429 14:04:40.085659 10389 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:04:40.085871 10389 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:04:40.086421 10930 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:40.086144 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40335 (local address 127.2.197.195:52249)
0429 14:04:40.086353 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:40.086390 (+ 37us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188}
I20260429 14:04:40.088886 10389 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 9782bdb83d3143c6a2f310e1c7de398f: Sending DeleteTablet for 3 replicas of tablet b918b04d3c334e4aa7dad1fc77e29c06
I20260429 14:04:40.089370 10517 tablet_service.cc:1558] Processing DeleteTablet for tablet b918b04d3c334e4aa7dad1fc77e29c06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:04:40 UTC) from {username='oryx', principal='oryx/127.2.197.252@KRBTEST.COM'} at 127.0.0.1:45576
I20260429 14:04:40.089649 10653 tablet_service.cc:1558] Processing DeleteTablet for tablet b918b04d3c334e4aa7dad1fc77e29c06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:04:40 UTC) from {username='oryx', principal='oryx/127.2.197.252@KRBTEST.COM'} at 127.0.0.1:52514
I20260429 14:04:40.089705 10790 tablet_service.cc:1558] Processing DeleteTablet for tablet b918b04d3c334e4aa7dad1fc77e29c06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:04:40 UTC) from {username='oryx', principal='oryx/127.2.197.252@KRBTEST.COM'} at 127.0.0.1:59428
I20260429 14:04:40.089934 10971 tablet_replica.cc:333] T b918b04d3c334e4aa7dad1fc77e29c06 P 468ecc83d41743dc981739ecb9b20067: stopping tablet replica
I20260429 14:04:40.089926 10789 tablet_service.cc:1558] Processing DeleteTablet for tablet b918b04d3c334e4aa7dad1fc77e29c06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:04:40 UTC) from {username='oryx', principal='oryx/127.2.197.252@KRBTEST.COM'} at 127.0.0.1:59428
I20260429 14:04:40.089998 10971 raft_consensus.cc:2243] T b918b04d3c334e4aa7dad1fc77e29c06 P 468ecc83d41743dc981739ecb9b20067 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:04:40.090051 10971 raft_consensus.cc:2272] T b918b04d3c334e4aa7dad1fc77e29c06 P 468ecc83d41743dc981739ecb9b20067 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:04:40.090433 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 10454
I20260429 14:04:40.092000 10972 tablet_replica.cc:333] T b918b04d3c334e4aa7dad1fc77e29c06 P 01baffeacfa6457094831efd2c408c4e: stopping tablet replica
I20260429 14:04:40.092056 10972 raft_consensus.cc:2243] T b918b04d3c334e4aa7dad1fc77e29c06 P 01baffeacfa6457094831efd2c408c4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:04:40.092096 10972 raft_consensus.cc:2272] T b918b04d3c334e4aa7dad1fc77e29c06 P 01baffeacfa6457094831efd2c408c4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:04:40.090051 10973 tablet_replica.cc:333] T b918b04d3c334e4aa7dad1fc77e29c06 P 3e025d72db7c49159529617974c08232: stopping tablet replica
I20260429 14:04:40.092223 10973 raft_consensus.cc:2243] T b918b04d3c334e4aa7dad1fc77e29c06 P 3e025d72db7c49159529617974c08232 [term 1 LEADER]: Raft consensus shutting down.
I20260429 14:04:40.092304 10973 raft_consensus.cc:2272] T b918b04d3c334e4aa7dad1fc77e29c06 P 3e025d72db7c49159529617974c08232 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:04:40.092489 10972 ts_tablet_manager.cc:1916] T b918b04d3c334e4aa7dad1fc77e29c06 P 01baffeacfa6457094831efd2c408c4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:04:40.094033 10972 ts_tablet_manager.cc:1929] T b918b04d3c334e4aa7dad1fc77e29c06 P 01baffeacfa6457094831efd2c408c4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 14:04:40.094080 10972 log.cc:1199] T b918b04d3c334e4aa7dad1fc77e29c06 P 01baffeacfa6457094831efd2c408c4e: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-1/wal/wals/b918b04d3c334e4aa7dad1fc77e29c06
I20260429 14:04:40.094410 10972 ts_tablet_manager.cc:1950] T b918b04d3c334e4aa7dad1fc77e29c06 P 01baffeacfa6457094831efd2c408c4e: Deleting consensus metadata
I20260429 14:04:40.095302 10375 catalog_manager.cc:5002] TS 01baffeacfa6457094831efd2c408c4e (127.2.197.194:44633): tablet b918b04d3c334e4aa7dad1fc77e29c06 (table test-table [id=1d5b0b6a75ea4cbdb16870522b9c812f]) successfully deleted
I20260429 14:04:40.096371 10973 ts_tablet_manager.cc:1916] T b918b04d3c334e4aa7dad1fc77e29c06 P 3e025d72db7c49159529617974c08232: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:04:40.097914 10946 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:40.097753 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:45535 (local address 127.0.0.1:45588)
0429 14:04:40.097824 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:40.097866 (+ 42us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.193:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260429 14:04:40.097995 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 10590
I20260429 14:04:40.099313 10973 ts_tablet_manager.cc:1929] T b918b04d3c334e4aa7dad1fc77e29c06 P 3e025d72db7c49159529617974c08232: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 14:04:40.099398 10973 log.cc:1199] T b918b04d3c334e4aa7dad1fc77e29c06 P 3e025d72db7c49159529617974c08232: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777471413505459-2839-0/minicluster-data/ts-2/wal/wals/b918b04d3c334e4aa7dad1fc77e29c06
I20260429 14:04:40.099689 10973 ts_tablet_manager.cc:1950] T b918b04d3c334e4aa7dad1fc77e29c06 P 3e025d72db7c49159529617974c08232: Deleting consensus metadata
I20260429 14:04:40.100127 10375 catalog_manager.cc:5002] TS 3e025d72db7c49159529617974c08232 (127.2.197.195:42039): tablet b918b04d3c334e4aa7dad1fc77e29c06 (table test-table [id=1d5b0b6a75ea4cbdb16870522b9c812f]) successfully deleted
W20260429 14:04:40.100308 10375 catalog_manager.cc:4969] TS 3e025d72db7c49159529617974c08232 (127.2.197.195:42039): delete failed for tablet b918b04d3c334e4aa7dad1fc77e29c06 because the tablet was not found. No further retry: Not found: Tablet not found: b918b04d3c334e4aa7dad1fc77e29c06
W20260429 14:04:40.100885 10377 catalog_manager.cc:4729] TS 468ecc83d41743dc981739ecb9b20067 (127.2.197.193:45535): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b918b04d3c334e4aa7dad1fc77e29c06: Network error: Client connection negotiation failed: client connection to 127.2.197.193:45535: connect: Connection refused (error 111)
I20260429 14:04:40.104765 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 10726
I20260429 14:04:40.111968 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 10058
I20260429 14:04:40.117453 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 10197
2026-04-29T14:04:40Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (8778 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:04:40 dist-test-slave-8n77 krb5kdc[10976](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:04:40 dist-test-slave-8n77 krb5kdc[10976](info): set up 2 sockets
Apr 29 14:04:40 dist-test-slave-8n77 krb5kdc[10976](info): commencing operation
krb5kdc: starting...
W20260429 14:04:42.184067 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.042s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:04:42 dist-test-slave-8n77 krb5kdc[10976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:04:42Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:04:42Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/krb5kdc/oryx.keytab.
I20260429 14:04:42.329020 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:42715
--webserver_interface=127.2.197.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.197.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:45147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:42715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:04:42.406387 10992 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:42.406590 10992 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:42.406628 10992 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:42.408210 10992 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:04:42.408289 10992 flags.cc:432] Enabled experimental flag: --txn_manager_enabled=true
W20260429 14:04:42.408316 10992 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:42.408349 10992 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:42.408375 10992 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:04:42.408401 10992 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:04:42.410053 10992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:45147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:42715
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.197.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:42715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:42.410357 10992 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:42.410548 10992 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:42.413704 11000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:42.413731 10992 server_base.cc:1061] running on GCE node
W20260429 14:04:42.413707 10998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:42.413705 10997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:42.414237 10992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:42.414548 10992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:42.415717 10992 hybrid_clock.cc:648] HybridClock initialized: now 1777471482415687 us; error 52 us; skew 500 ppm
Apr 29 14:04:42 dist-test-slave-8n77 krb5kdc[10976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471482, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:42.418377 10992 init.cc:377] Logged in from keytab as oryx/127.2.197.254@KRBTEST.COM (short username oryx)
I20260429 14:04:42.419044 10992 webserver.cc:492] Webserver started at http://127.2.197.254:34505/ using document root <none> and password file <none>
I20260429 14:04:42.419255 10992 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:42.419320 10992 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:42.419436 10992 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:04:42.420379 10992 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "bb7ccf4eb15e4d9e84fe3634fa4b154b"
format_stamp: "Formatted at 2026-04-29 14:04:42 on dist-test-slave-8n77"
server_key: "524faca78ff058de8fb4bd3619ed9c67"
server_key_iv: "4ee806117c99151c0570116349b32920"
server_key_version: "encryptionkey@0"
I20260429 14:04:42.420725 10992 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "bb7ccf4eb15e4d9e84fe3634fa4b154b"
format_stamp: "Formatted at 2026-04-29 14:04:42 on dist-test-slave-8n77"
server_key: "524faca78ff058de8fb4bd3619ed9c67"
server_key_iv: "4ee806117c99151c0570116349b32920"
server_key_version: "encryptionkey@0"
I20260429 14:04:42.422065 10992 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260429 14:04:42.423048 11007 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:42.423355 10992 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:42.423425 10992 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "bb7ccf4eb15e4d9e84fe3634fa4b154b"
format_stamp: "Formatted at 2026-04-29 14:04:42 on dist-test-slave-8n77"
server_key: "524faca78ff058de8fb4bd3619ed9c67"
server_key_iv: "4ee806117c99151c0570116349b32920"
server_key_version: "encryptionkey@0"
I20260429 14:04:42.423475 10992 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:42.433594 10992 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:42.435837 10992 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:42.435956 10992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:42.440922 10992 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:42715
I20260429 14:04:42.441049 11069 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:42715 every 8 connection(s)
I20260429 14:04:42.441309 10992 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:04:42.442170 11070 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:42.443442 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 10992
I20260429 14:04:42.443531 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:04:42.443706 2839 external_mini_cluster.cc:1468] Setting key 7865868da5da72f4a59e971c33c7b64d
I20260429 14:04:42.444873 11070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb7ccf4eb15e4d9e84fe3634fa4b154b: Bootstrap starting.
I20260429 14:04:42.445662 11070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb7ccf4eb15e4d9e84fe3634fa4b154b: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:42.445948 11070 log.cc:826] T 00000000000000000000000000000000 P bb7ccf4eb15e4d9e84fe3634fa4b154b: Log is configured to *not* fsync() on all Append() calls
I20260429 14:04:42.446717 11070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb7ccf4eb15e4d9e84fe3634fa4b154b: No bootstrap required, opened a new log
I20260429 14:04:42.448369 11070 raft_consensus.cc:359] T 00000000000000000000000000000000 P bb7ccf4eb15e4d9e84fe3634fa4b154b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb7ccf4eb15e4d9e84fe3634fa4b154b" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 42715 } }
I20260429 14:04:42.448537 11070 raft_consensus.cc:385] T 00000000000000000000000000000000 P bb7ccf4eb15e4d9e84fe3634fa4b154b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:42.448575 11070 raft_consensus.cc:740] T 00000000000000000000000000000000 P bb7ccf4eb15e4d9e84fe3634fa4b154b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb7ccf4eb15e4d9e84fe3634fa4b154b, State: Initialized, Role: FOLLOWER
I20260429 14:04:42.448714 11070 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb7ccf4eb15e4d9e84fe3634fa4b154b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb7ccf4eb15e4d9e84fe3634fa4b154b" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 42715 } }
I20260429 14:04:42.448804 11070 raft_consensus.cc:399] T 00000000000000000000000000000000 P bb7ccf4eb15e4d9e84fe3634fa4b154b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:04:42.448839 11070 raft_consensus.cc:493] T 00000000000000000000000000000000 P bb7ccf4eb15e4d9e84fe3634fa4b154b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:04:42.448899 11070 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bb7ccf4eb15e4d9e84fe3634fa4b154b [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:42.449658 11070 raft_consensus.cc:515] T 00000000000000000000000000000000 P bb7ccf4eb15e4d9e84fe3634fa4b154b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb7ccf4eb15e4d9e84fe3634fa4b154b" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 42715 } }
I20260429 14:04:42.449762 11070 leader_election.cc:304] T 00000000000000000000000000000000 P bb7ccf4eb15e4d9e84fe3634fa4b154b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb7ccf4eb15e4d9e84fe3634fa4b154b; no voters:
I20260429 14:04:42.449958 11070 leader_election.cc:290] T 00000000000000000000000000000000 P bb7ccf4eb15e4d9e84fe3634fa4b154b [CANDIDATE]: Term 1 election: Requested vote from peers
Apr 29 14:04:42 dist-test-slave-8n77 krb5kdc[10976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM
I20260429 14:04:42.450150 11075 raft_consensus.cc:2804] T 00000000000000000000000000000000 P bb7ccf4eb15e4d9e84fe3634fa4b154b [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:04:42.450196 11070 sys_catalog.cc:565] T 00000000000000000000000000000000 P bb7ccf4eb15e4d9e84fe3634fa4b154b [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:04:42.450317 11075 raft_consensus.cc:697] T 00000000000000000000000000000000 P bb7ccf4eb15e4d9e84fe3634fa4b154b [term 1 LEADER]: Becoming Leader. State: Replica: bb7ccf4eb15e4d9e84fe3634fa4b154b, State: Running, Role: LEADER
I20260429 14:04:42.450479 11075 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb7ccf4eb15e4d9e84fe3634fa4b154b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb7ccf4eb15e4d9e84fe3634fa4b154b" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 42715 } }
I20260429 14:04:42.450989 11075 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb7ccf4eb15e4d9e84fe3634fa4b154b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb7ccf4eb15e4d9e84fe3634fa4b154b. Latest consensus state: current_term: 1 leader_uuid: "bb7ccf4eb15e4d9e84fe3634fa4b154b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb7ccf4eb15e4d9e84fe3634fa4b154b" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 42715 } } }
I20260429 14:04:42.451165 11075 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb7ccf4eb15e4d9e84fe3634fa4b154b [sys.catalog]: This master's current role is: LEADER
I20260429 14:04:42.451388 11084 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:04:42.451476 11077 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb7ccf4eb15e4d9e84fe3634fa4b154b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb7ccf4eb15e4d9e84fe3634fa4b154b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb7ccf4eb15e4d9e84fe3634fa4b154b" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 42715 } } }
I20260429 14:04:42.451857 11077 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb7ccf4eb15e4d9e84fe3634fa4b154b [sys.catalog]: This master's current role is: LEADER
I20260429 14:04:42.451897 11084 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:04:42.453665 11084 catalog_manager.cc:1357] Generated new cluster ID: 48b1eb90cd1c42a4999a6c49f5ae3504
I20260429 14:04:42.453715 11084 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:04:42.454476 11073 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:42.444300 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57812 (local address 127.2.197.254:42715)
0429 14:04:42.444471 (+ 171us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:42.444480 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:42.444499 (+ 19us) server_negotiation.cc:408] Connection header received
0429 14:04:42.444811 (+ 312us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:42.444817 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:42.444855 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:42.445026 (+ 171us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:42.446093 (+ 1067us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:42.446912 (+ 819us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:42.447705 (+ 793us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:42.447962 (+ 257us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:42.450612 (+ 2650us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:42.450620 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:42.450621 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:42.450635 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:42.452910 (+ 2275us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:42.453327 (+ 417us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:42.453331 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:42.453331 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:42.453392 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:42.453611 (+ 219us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:42.453613 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:42.453613 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:42.453857 (+ 244us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:42.454008 (+ 151us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:42.454159 (+ 151us) server_negotiation.cc:300] Negotiation successful
0429 14:04:42.454320 (+ 161us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":102,"threads_started":1}
I20260429 14:04:42.476022 11084 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:04:42.476866 11084 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:04:42.485824 11084 catalog_manager.cc:6044] T 00000000000000000000000000000000 P bb7ccf4eb15e4d9e84fe3634fa4b154b: Generated new TSK 0
I20260429 14:04:42.486035 11084 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/krb5kdc/oryx.keytab.
I20260429 14:04:42.549333 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.193:0
--local_ip_for_outbound_sockets=127.2.197.193
--webserver_interface=127.2.197.193
--webserver_port=0
--tserver_master_addrs=127.2.197.254:42715
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.197.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:45147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:04:42.627887 11098 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:42.628067 11098 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:42.628093 11098 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:42.629647 11098 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:42.629700 11098 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:42.629730 11098 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260429 14:04:42.629750 11098 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.193
I20260429 14:04:42.631532 11098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:45147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.197.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.197.193
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.197.254:42715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.193
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:42.631783 11098 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:42.632009 11098 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:42.634789 11104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:42.634836 11098 server_base.cc:1061] running on GCE node
W20260429 14:04:42.634891 11106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:42.634815 11103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:42.635138 11098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:42.635327 11098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:42.636458 11098 hybrid_clock.cc:648] HybridClock initialized: now 1777471482636438 us; error 37 us; skew 500 ppm
Apr 29 14:04:42 dist-test-slave-8n77 krb5kdc[10976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471482, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:42.639081 11098 init.cc:377] Logged in from keytab as oryx/127.2.197.193@KRBTEST.COM (short username oryx)
I20260429 14:04:42.639674 11098 webserver.cc:492] Webserver started at http://127.2.197.193:34097/ using document root <none> and password file <none>
I20260429 14:04:42.639871 11098 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:42.639932 11098 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:42.640036 11098 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:04:42.641010 11098 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-0/data/instance:
uuid: "49573166d38746089e79505c86c6e93a"
format_stamp: "Formatted at 2026-04-29 14:04:42 on dist-test-slave-8n77"
server_key: "1c8195b1d7424fb4aebf4361b8346f75"
server_key_iv: "da8bb445637694abf4f2ecce93bb3cd1"
server_key_version: "encryptionkey@0"
I20260429 14:04:42.641335 11098 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance:
uuid: "49573166d38746089e79505c86c6e93a"
format_stamp: "Formatted at 2026-04-29 14:04:42 on dist-test-slave-8n77"
server_key: "1c8195b1d7424fb4aebf4361b8346f75"
server_key_iv: "da8bb445637694abf4f2ecce93bb3cd1"
server_key_version: "encryptionkey@0"
I20260429 14:04:42.642643 11098 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260429 14:04:42.643451 11113 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:42.643702 11098 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:04:42.643783 11098 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-0/wal
uuid: "49573166d38746089e79505c86c6e93a"
format_stamp: "Formatted at 2026-04-29 14:04:42 on dist-test-slave-8n77"
server_key: "1c8195b1d7424fb4aebf4361b8346f75"
server_key_iv: "da8bb445637694abf4f2ecce93bb3cd1"
server_key_version: "encryptionkey@0"
I20260429 14:04:42.643870 11098 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:42.669487 11098 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:42.671787 11098 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:42.671976 11098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:42.672711 11098 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:04:42.672765 11098 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:42.672814 11098 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:04:42.672829 11098 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 29 14:04:42 dist-test-slave-8n77 krb5kdc[10976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471482, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM
I20260429 14:04:42.680431 11098 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.193:42967
I20260429 14:04:42.680524 11228 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.193:42967 every 8 connection(s)
I20260429 14:04:42.680819 11098 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
I20260429 14:04:42.682140 11073 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:42.673236 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:46037 (local address 127.2.197.254:42715)
0429 14:04:42.673315 (+ 79us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:42.673318 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:42.673482 (+ 164us) server_negotiation.cc:408] Connection header received
0429 14:04:42.674026 (+ 544us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:42.674028 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:42.674042 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:42.674125 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:42.675072 (+ 947us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:42.675933 (+ 861us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:42.676525 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:42.676708 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:42.678969 (+ 2261us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:42.678974 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:42.678974 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:42.678983 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:42.680780 (+ 1797us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:42.681293 (+ 513us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:42.681294 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:42.681294 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:42.681350 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:42.681619 (+ 269us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:42.681619 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:42.681620 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:42.681714 (+ 94us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:42.681769 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:42.681994 (+ 225us) server_negotiation.cc:300] Negotiation successful
0429 14:04:42.682075 (+ 81us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260429 14:04:42.682196 11122 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:42.673199 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:42715 (local address 127.2.197.193:46037)
0429 14:04:42.673358 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:42.673389 (+ 31us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:42.673927 (+ 538us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:42.674224 (+ 297us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:42.674227 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:42.674719 (+ 492us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:42.674986 (+ 267us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:42.674989 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:42.676030 (+ 1041us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:42.676032 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:42.676433 (+ 401us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:42.676435 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:42.676580 (+ 145us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:42.677156 (+ 576us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:42.677170 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:42.678870 (+ 1700us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:42.680912 (+ 2042us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:42.680919 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:42.680922 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:42.681222 (+ 300us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:42.681453 (+ 231us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:42.681454 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:42.681455 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:42.681560 (+ 105us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:42.681778 (+ 218us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:42.681782 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:42.681884 (+ 102us) client_negotiation.cc:770] Sending connection context
0429 14:04:42.681966 (+ 82us) client_negotiation.cc:241] Negotiation successful
0429 14:04:42.682074 (+ 108us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":91,"threads_started":1}
I20260429 14:04:42.682718 11229 heartbeater.cc:344] Connected to a master server at 127.2.197.254:42715
I20260429 14:04:42.682807 11229 heartbeater.cc:461] Registering TS with master...
I20260429 14:04:42.683004 11229 heartbeater.cc:507] Master 127.2.197.254:42715 requested a full tablet report, sending...
I20260429 14:04:42.683516 11024 ts_manager.cc:194] Registered new tserver with Master: 49573166d38746089e79505c86c6e93a (127.2.197.193:42967)
I20260429 14:04:42.684156 11024 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.197.193@KRBTEST.COM'} at 127.2.197.193:46037
I20260429 14:04:42.684733 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 11098
I20260429 14:04:42.684849 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance
I20260429 14:04:42.685052 2839 external_mini_cluster.cc:1468] Setting key 36abbf9bfd68659e8495694b921e455f
I20260429 14:04:42.690423 11073 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:42.684982 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:41241 (local address 127.2.197.254:42715)
0429 14:04:42.685038 (+ 56us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:42.685039 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:42.685046 (+ 7us) server_negotiation.cc:408] Connection header received
0429 14:04:42.685062 (+ 16us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:42.685063 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:42.685071 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:42.685134 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:42.685628 (+ 494us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:42.686152 (+ 524us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:42.686900 (+ 748us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:42.687107 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:42.687833 (+ 726us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:42.687836 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:42.687836 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:42.687842 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:42.689152 (+ 1310us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:42.689622 (+ 470us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:42.689623 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:42.689624 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:42.689648 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:42.689867 (+ 219us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:42.689869 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:42.689869 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:42.689968 (+ 99us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:42.690034 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:42.690221 (+ 187us) server_negotiation.cc:300] Negotiation successful
0429 14:04:42.690359 (+ 138us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260429 14:04:42.690512 11235 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:42.684464 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:42715 (local address 127.2.197.193:41241)
0429 14:04:42.684620 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:42.684630 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:42.684691 (+ 61us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:42.685154 (+ 463us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:42.685156 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:42.685338 (+ 182us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:42.685515 (+ 177us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:42.685517 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:42.686235 (+ 718us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:42.686236 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:42.686763 (+ 527us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:42.686765 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:42.686875 (+ 110us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:42.687361 (+ 486us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:42.687367 (+ 6us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:42.687735 (+ 368us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:42.689235 (+ 1500us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:42.689237 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:42.689237 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:42.689462 (+ 225us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:42.689733 (+ 271us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:42.689734 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:42.689735 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:42.689774 (+ 39us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:42.690047 (+ 273us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:42.690049 (+ 2us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:42.690113 (+ 64us) client_negotiation.cc:770] Sending connection context
0429 14:04:42.690197 (+ 84us) client_negotiation.cc:241] Negotiation successful
0429 14:04:42.690352 (+ 155us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for oryx/127.2.197.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.2.197.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/krb5kdc/oryx.keytab.
I20260429 14:04:42.743052 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.194:0
--local_ip_for_outbound_sockets=127.2.197.194
--webserver_interface=127.2.197.194
--webserver_port=0
--tserver_master_addrs=127.2.197.254:42715
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.197.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:45147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:04:42.822252 11240 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:42.822515 11240 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:42.822539 11240 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:42.823998 11240 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:42.824045 11240 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:42.824066 11240 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260429 14:04:42.824083 11240 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.194
I20260429 14:04:42.825673 11240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:45147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.197.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.197.194
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.197.254:42715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.194
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:42.825861 11240 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:42.825999 11240 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:42.828558 11248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:42.828558 11245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:42.828584 11246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:42.828841 11240 server_base.cc:1061] running on GCE node
I20260429 14:04:42.829033 11240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:42.829203 11240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:42.830353 11240 hybrid_clock.cc:648] HybridClock initialized: now 1777471482830336 us; error 27 us; skew 500 ppm
Apr 29 14:04:42 dist-test-slave-8n77 krb5kdc[10976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471482, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:42.832778 11240 init.cc:377] Logged in from keytab as oryx/127.2.197.194@KRBTEST.COM (short username oryx)
I20260429 14:04:42.833302 11240 webserver.cc:492] Webserver started at http://127.2.197.194:37461/ using document root <none> and password file <none>
I20260429 14:04:42.833464 11240 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:42.833510 11240 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:42.833580 11240 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:04:42.834568 11240 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-1/data/instance:
uuid: "934c8c1fa4f64ba7941168f5ab504ce7"
format_stamp: "Formatted at 2026-04-29 14:04:42 on dist-test-slave-8n77"
server_key: "eafb56428129082f011e4652c16cefdc"
server_key_iv: "e53d80201b25b570de0c4b6b44e76870"
server_key_version: "encryptionkey@0"
I20260429 14:04:42.834846 11240 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance:
uuid: "934c8c1fa4f64ba7941168f5ab504ce7"
format_stamp: "Formatted at 2026-04-29 14:04:42 on dist-test-slave-8n77"
server_key: "eafb56428129082f011e4652c16cefdc"
server_key_iv: "e53d80201b25b570de0c4b6b44e76870"
server_key_version: "encryptionkey@0"
I20260429 14:04:42.836174 11240 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:42.836959 11255 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:42.837184 11240 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:42.837260 11240 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-1/wal
uuid: "934c8c1fa4f64ba7941168f5ab504ce7"
format_stamp: "Formatted at 2026-04-29 14:04:42 on dist-test-slave-8n77"
server_key: "eafb56428129082f011e4652c16cefdc"
server_key_iv: "e53d80201b25b570de0c4b6b44e76870"
server_key_version: "encryptionkey@0"
I20260429 14:04:42.837316 11240 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:42.853271 11240 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:42.855394 11240 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:42.855513 11240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:42.856034 11240 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:04:42.856072 11240 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:42.856096 11240 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:04:42.856158 11240 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:42.862864 11240 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.194:37361
I20260429 14:04:42.862959 11370 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.194:37361 every 8 connection(s)
I20260429 14:04:42.863240 11240 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
Apr 29 14:04:42 dist-test-slave-8n77 krb5kdc[10976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471482, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM
I20260429 14:04:42.867684 11073 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:42.856904 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:54283 (local address 127.2.197.254:42715)
0429 14:04:42.856988 (+ 84us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:42.856991 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:42.857171 (+ 180us) server_negotiation.cc:408] Connection header received
0429 14:04:42.857884 (+ 713us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:42.857885 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:42.857900 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:42.857983 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:42.859061 (+ 1078us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:42.859909 (+ 848us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:42.860833 (+ 924us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:42.860994 (+ 161us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:42.864558 (+ 3564us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:42.864569 (+ 11us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:42.864569 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:42.864578 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:42.866049 (+ 1471us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:42.866727 (+ 678us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:42.866728 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:42.866729 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:42.866756 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:42.866989 (+ 233us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:42.866990 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:42.866990 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:42.867103 (+ 113us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:42.867158 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:42.867520 (+ 362us) server_negotiation.cc:300] Negotiation successful
0429 14:04:42.867613 (+ 93us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260429 14:04:42.868175 11264 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:42.856867 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:42715 (local address 127.2.197.194:54283)
0429 14:04:42.857062 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:42.857089 (+ 27us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:42.857767 (+ 678us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:42.858107 (+ 340us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:42.858112 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:42.858557 (+ 445us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:42.858969 (+ 412us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:42.858974 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:42.860019 (+ 1045us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:42.860021 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:42.860728 (+ 707us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:42.860731 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:42.860888 (+ 157us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:42.861535 (+ 647us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:42.861548 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:42.864361 (+ 2813us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:42.866210 (+ 1849us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:42.866218 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:42.866220 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:42.866607 (+ 387us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:42.866852 (+ 245us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:42.866853 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:42.866853 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:42.866919 (+ 66us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:42.867206 (+ 287us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:42.867211 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:42.867344 (+ 133us) client_negotiation.cc:770] Sending connection context
0429 14:04:42.867448 (+ 104us) client_negotiation.cc:241] Negotiation successful
0429 14:04:42.867788 (+ 340us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":91,"threads_started":1}
I20260429 14:04:42.868341 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 11240
I20260429 14:04:42.868431 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance
I20260429 14:04:42.868685 2839 external_mini_cluster.cc:1468] Setting key c0d17c68ab0322052b346c78eb46c5f6
I20260429 14:04:42.868841 11371 heartbeater.cc:344] Connected to a master server at 127.2.197.254:42715
I20260429 14:04:42.868940 11371 heartbeater.cc:461] Registering TS with master...
I20260429 14:04:42.869626 11371 heartbeater.cc:507] Master 127.2.197.254:42715 requested a full tablet report, sending...
I20260429 14:04:42.870002 11023 ts_manager.cc:194] Registered new tserver with Master: 934c8c1fa4f64ba7941168f5ab504ce7 (127.2.197.194:37361)
I20260429 14:04:42.870597 11023 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.197.194@KRBTEST.COM'} at 127.2.197.194:54283
I20260429 14:04:42.876174 11073 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:42.869906 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:52149 (local address 127.2.197.254:42715)
0429 14:04:42.869943 (+ 37us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:42.869944 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:42.870239 (+ 295us) server_negotiation.cc:408] Connection header received
0429 14:04:42.870339 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:42.870341 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:42.870352 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:42.870407 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:42.871207 (+ 800us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:42.871996 (+ 789us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:42.872685 (+ 689us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:42.872880 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:42.873636 (+ 756us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:42.873643 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:42.873644 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:42.873649 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:42.875097 (+ 1448us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:42.875490 (+ 393us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:42.875491 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:42.875492 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:42.875514 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:42.875690 (+ 176us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:42.875690 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:42.875691 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:42.875809 (+ 118us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:42.875862 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:42.876038 (+ 176us) server_negotiation.cc:300] Negotiation successful
0429 14:04:42.876107 (+ 69us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260429 14:04:42.876250 11377 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:42.869984 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:42715 (local address 127.2.197.194:52149)
0429 14:04:42.870166 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:42.870175 (+ 9us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:42.870252 (+ 77us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:42.870412 (+ 160us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:42.870413 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:42.870578 (+ 165us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:42.871091 (+ 513us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:42.871095 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:42.872091 (+ 996us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:42.872091 (+ 0us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:42.872574 (+ 483us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:42.872576 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:42.872688 (+ 112us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:42.873259 (+ 571us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:42.873263 (+ 4us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:42.873518 (+ 255us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:42.875190 (+ 1672us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:42.875191 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:42.875192 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:42.875401 (+ 209us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:42.875582 (+ 181us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:42.875583 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:42.875583 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:42.875622 (+ 39us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:42.875879 (+ 257us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:42.875880 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:42.875934 (+ 54us) client_negotiation.cc:770] Sending connection context
0429 14:04:42.876015 (+ 81us) client_negotiation.cc:241] Negotiation successful
0429 14:04:42.876131 (+ 116us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":91,"threads_started":1}
WARNING: no policy specified for oryx/127.2.197.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.2.197.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/krb5kdc/oryx.keytab.
I20260429 14:04:42.926923 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.195:0
--local_ip_for_outbound_sockets=127.2.197.195
--webserver_interface=127.2.197.195
--webserver_port=0
--tserver_master_addrs=127.2.197.254:42715
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.197.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:45147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:04:43.003882 11382 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:43.004101 11382 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:43.004138 11382 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:43.005671 11382 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:43.005764 11382 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:43.005802 11382 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260429 14:04:43.005842 11382 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.195
I20260429 14:04:43.007629 11382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:45147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.197.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.197.195
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.197.254:42715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.195
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:43.007902 11382 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:43.008097 11382 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:43.010581 11388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:43.010604 11390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:43.010613 11387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:43.011056 11382 server_base.cc:1061] running on GCE node
I20260429 14:04:43.011284 11382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:43.011505 11382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:43.012641 11382 hybrid_clock.cc:648] HybridClock initialized: now 1777471483012627 us; error 29 us; skew 500 ppm
Apr 29 14:04:43 dist-test-slave-8n77 krb5kdc[10976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471483, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:43.015587 11382 init.cc:377] Logged in from keytab as oryx/127.2.197.195@KRBTEST.COM (short username oryx)
I20260429 14:04:43.016175 11382 webserver.cc:492] Webserver started at http://127.2.197.195:46549/ using document root <none> and password file <none>
I20260429 14:04:43.016381 11382 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:43.016443 11382 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:43.016551 11382 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:04:43.017496 11382 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-2/data/instance:
uuid: "f49e2dd467f042f08da4345b01c63e9b"
format_stamp: "Formatted at 2026-04-29 14:04:43 on dist-test-slave-8n77"
server_key: "dbb85670de8804337061da2918206e69"
server_key_iv: "e1cff4c921bcb97d96c77b60b8c2e632"
server_key_version: "encryptionkey@0"
I20260429 14:04:43.017843 11382 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance:
uuid: "f49e2dd467f042f08da4345b01c63e9b"
format_stamp: "Formatted at 2026-04-29 14:04:43 on dist-test-slave-8n77"
server_key: "dbb85670de8804337061da2918206e69"
server_key_iv: "e1cff4c921bcb97d96c77b60b8c2e632"
server_key_version: "encryptionkey@0"
I20260429 14:04:43.019212 11382 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260429 14:04:43.020097 11397 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:43.020354 11382 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:04:43.020442 11382 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-2/wal
uuid: "f49e2dd467f042f08da4345b01c63e9b"
format_stamp: "Formatted at 2026-04-29 14:04:43 on dist-test-slave-8n77"
server_key: "dbb85670de8804337061da2918206e69"
server_key_iv: "e1cff4c921bcb97d96c77b60b8c2e632"
server_key_version: "encryptionkey@0"
I20260429 14:04:43.020517 11382 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:43.030853 11382 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:43.032809 11382 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:43.032965 11382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:43.033530 11382 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:04:43.033591 11382 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:43.033629 11382 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:04:43.033658 11382 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 29 14:04:43 dist-test-slave-8n77 krb5kdc[10976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471483, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.195@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM
I20260429 14:04:43.041189 11382 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.195:37495
I20260429 14:04:43.041258 11512 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.195:37495 every 8 connection(s)
I20260429 14:04:43.041538 11382 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
I20260429 14:04:43.042526 11073 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:43.033891 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:51823 (local address 127.2.197.254:42715)
0429 14:04:43.033942 (+ 51us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:43.033945 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:43.034197 (+ 252us) server_negotiation.cc:408] Connection header received
0429 14:04:43.034974 (+ 777us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:43.034975 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:43.034988 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:43.035085 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:43.035845 (+ 760us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.036309 (+ 464us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:43.036862 (+ 553us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.037038 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:43.039205 (+ 2167us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:43.039211 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:43.039211 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:43.039219 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:43.040920 (+ 1701us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:43.041461 (+ 541us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:43.041462 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:43.041463 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:43.041489 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:43.041687 (+ 198us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:43.041688 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:43.041689 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:43.041909 (+ 220us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:43.042028 (+ 119us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:43.042341 (+ 313us) server_negotiation.cc:300] Negotiation successful
0429 14:04:43.042432 (+ 91us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260429 14:04:43.042646 11406 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:43.033894 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:42715 (local address 127.2.197.195:51823)
0429 14:04:43.034067 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:43.034107 (+ 40us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:43.034828 (+ 721us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:43.035151 (+ 323us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:43.035155 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:43.035418 (+ 263us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:43.035719 (+ 301us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:43.035722 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.036397 (+ 675us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:43.036398 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:43.036779 (+ 381us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:43.036781 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.036904 (+ 123us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:43.037408 (+ 504us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:43.037416 (+ 8us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:43.039102 (+ 1686us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:43.041060 (+ 1958us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:43.041065 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:43.041067 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:43.041392 (+ 325us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:43.041555 (+ 163us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:43.041555 (+ 0us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:43.041556 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:43.041619 (+ 63us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:43.042059 (+ 440us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:43.042063 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:43.042176 (+ 113us) client_negotiation.cc:770] Sending connection context
0429 14:04:43.042281 (+ 105us) client_negotiation.cc:241] Negotiation successful
0429 14:04:43.042438 (+ 157us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
I20260429 14:04:43.043294 11513 heartbeater.cc:344] Connected to a master server at 127.2.197.254:42715
I20260429 14:04:43.043382 11513 heartbeater.cc:461] Registering TS with master...
I20260429 14:04:43.043581 11513 heartbeater.cc:507] Master 127.2.197.254:42715 requested a full tablet report, sending...
I20260429 14:04:43.044373 11023 ts_manager.cc:194] Registered new tserver with Master: f49e2dd467f042f08da4345b01c63e9b (127.2.197.195:37495)
I20260429 14:04:43.044898 11023 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.197.195@KRBTEST.COM'} at 127.2.197.195:51823
I20260429 14:04:43.050482 11073 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:43.044550 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:41975 (local address 127.2.197.254:42715)
0429 14:04:43.044617 (+ 67us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:43.044619 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:43.044771 (+ 152us) server_negotiation.cc:408] Connection header received
0429 14:04:43.044796 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:43.044797 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:43.044807 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:43.044876 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:43.045388 (+ 512us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.046036 (+ 648us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:43.046768 (+ 732us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.046959 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:43.047773 (+ 814us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:43.047777 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:43.047778 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:43.047785 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:43.049122 (+ 1337us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:43.049492 (+ 370us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:43.049493 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:43.049494 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:43.049539 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:43.049867 (+ 328us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:43.049868 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:43.049869 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:43.049992 (+ 123us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:43.050078 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:43.050301 (+ 223us) server_negotiation.cc:300] Negotiation successful
0429 14:04:43.050389 (+ 88us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260429 14:04:43.050540 11519 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:43.044489 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:42715 (local address 127.2.197.195:41975)
0429 14:04:43.044674 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:43.044689 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:43.044747 (+ 58us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:43.044915 (+ 168us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:43.044916 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:43.045130 (+ 214us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:43.045295 (+ 165us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:43.045297 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.046151 (+ 854us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:43.046153 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:43.046675 (+ 522us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:43.046677 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.046771 (+ 94us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:43.047355 (+ 584us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:43.047360 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:43.047656 (+ 296us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:43.049232 (+ 1576us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:43.049233 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:43.049234 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:43.049420 (+ 186us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:43.049633 (+ 213us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:43.049634 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:43.049634 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:43.049665 (+ 31us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:43.050066 (+ 401us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:43.050067 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:43.050145 (+ 78us) client_negotiation.cc:770] Sending connection context
0429 14:04:43.050232 (+ 87us) client_negotiation.cc:241] Negotiation successful
0429 14:04:43.050377 (+ 145us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":94,"threads_started":1}
I20260429 14:04:43.051942 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 11382
I20260429 14:04:43.052028 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance
I20260429 14:04:43.052233 2839 external_mini_cluster.cc:1468] Setting key f1927c5af4a22e195a4bf003320a4443
I20260429 14:04:43.056098 2839 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Apr 29 14:04:43 dist-test-slave-8n77 krb5kdc[10976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471483, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Apr 29 14:04:43 dist-test-slave-8n77 krb5kdc[10976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:04:43 dist-test-slave-8n77 krb5kdc[10976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
W20260429 14:04:43.079712 11073 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:43.073025 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57824 (local address 127.2.197.254:42715)
0429 14:04:43.073101 (+ 76us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:43.073103 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:43.073112 (+ 9us) server_negotiation.cc:408] Connection header received
0429 14:04:43.073137 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:43.073138 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:43.073148 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:43.073211 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:43.074021 (+ 810us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.074899 (+ 878us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:43.075766 (+ 867us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.075917 (+ 151us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:43.079618 (+ 3701us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57824: BlockingRecv error: recv got EOF from 127.0.0.1:57824 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Apr 29 14:04:43 dist-test-slave-8n77 krb5kdc[10976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471483, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM
I20260429 14:04:43.088279 11073 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:43.080816 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57826 (local address 127.2.197.254:42715)
0429 14:04:43.080879 (+ 63us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:43.080881 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:43.080908 (+ 27us) server_negotiation.cc:408] Connection header received
0429 14:04:43.080980 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:43.080980 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:43.080990 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:43.081048 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:43.081697 (+ 649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.082180 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:43.082865 (+ 685us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.082992 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:43.085509 (+ 2517us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:43.085513 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:43.085514 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:43.085521 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:43.087023 (+ 1502us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:43.087390 (+ 367us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:43.087391 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:43.087392 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:43.087419 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:43.087647 (+ 228us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:43.087648 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:43.087648 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:43.087761 (+ 113us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:43.087875 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:43.088135 (+ 260us) server_negotiation.cc:300] Negotiation successful
0429 14:04:43.088206 (+ 71us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260429 14:04:43.090230 11023 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 14:04:43.091619 11023 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 14:04:43.098102 11538 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:43.094350 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:37361 (local address 127.0.0.1:38694)
0429 14:04:43.094569 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:43.094579 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:43.094642 (+ 63us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:43.095189 (+ 547us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:43.095189 (+ 0us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:43.095191 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:04:43.095431 (+ 240us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:43.095433 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.096797 (+ 1364us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:43.096797 (+ 0us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:43.097785 (+ 988us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:43.097787 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.097878 (+ 91us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:43.097913 (+ 35us) client_negotiation.cc:770] Sending connection context
0429 14:04:43.097964 (+ 51us) client_negotiation.cc:241] Negotiation successful
0429 14:04:43.098009 (+ 45us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":98,"threads_started":1}
I20260429 14:04:43.098387 11535 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:43.094133 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:42967 (local address 127.0.0.1:39604)
0429 14:04:43.094503 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:43.094537 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:43.094615 (+ 78us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:43.095019 (+ 404us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:43.095022 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:43.095025 (+ 3us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:04:43.095280 (+ 255us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:43.095282 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.096783 (+ 1501us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:43.096784 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:43.097810 (+ 1026us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:43.097812 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.097874 (+ 62us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:43.097913 (+ 39us) client_negotiation.cc:770] Sending connection context
0429 14:04:43.097964 (+ 51us) client_negotiation.cc:241] Negotiation successful
0429 14:04:43.098009 (+ 45us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":51,"threads_started":1}
I20260429 14:04:43.098605 11540 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:43.094743 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:38694 (local address 127.2.197.194:37361)
0429 14:04:43.094934 (+ 191us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:43.094945 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:43.094955 (+ 10us) server_negotiation.cc:408] Connection header received
0429 14:04:43.094992 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:43.094994 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:43.095009 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:43.095102 (+ 93us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:04:43.095517 (+ 415us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.096699 (+ 1182us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:43.097898 (+ 1199us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.098407 (+ 509us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:43.098462 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:43.098495 (+ 33us) server_negotiation.cc:300] Negotiation successful
0429 14:04:43.098535 (+ 40us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":91,"threads_started":1}
I20260429 14:04:43.098907 11536 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:43.094263 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:39604 (local address 127.2.197.193:42967)
0429 14:04:43.094561 (+ 298us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:43.094569 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:43.094587 (+ 18us) server_negotiation.cc:408] Connection header received
0429 14:04:43.094749 (+ 162us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:43.094753 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:43.094774 (+ 21us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:43.094869 (+ 95us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:04:43.095368 (+ 499us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.096635 (+ 1267us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:43.098069 (+ 1434us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.098726 (+ 657us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:43.098770 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:43.098798 (+ 28us) server_negotiation.cc:300] Negotiation successful
0429 14:04:43.098857 (+ 59us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":161,"threads_started":1}
I20260429 14:04:43.099118 11305 tablet_service.cc:1511] Processing CreateTablet for tablet 36e78fa7c12d4ad3920e31c608b0cc83 (DEFAULT_TABLE table=test-table [id=758d4d8eaf494f5c91aa954f927f1097]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:04:43.099416 11305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36e78fa7c12d4ad3920e31c608b0cc83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:43.099511 11537 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:43.094380 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:37495 (local address 127.0.0.1:37490)
0429 14:04:43.094741 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:43.094751 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:43.094793 (+ 42us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:43.095185 (+ 392us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:43.095186 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:43.095188 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:04:43.095431 (+ 243us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:43.095433 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.097752 (+ 2319us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:43.097753 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:43.099063 (+ 1310us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:43.099067 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.099365 (+ 298us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:43.099370 (+ 5us) client_negotiation.cc:770] Sending connection context
0429 14:04:43.099404 (+ 34us) client_negotiation.cc:241] Negotiation successful
0429 14:04:43.099452 (+ 48us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":73,"threads_started":1}
I20260429 14:04:43.099462 11163 tablet_service.cc:1511] Processing CreateTablet for tablet 36e78fa7c12d4ad3920e31c608b0cc83 (DEFAULT_TABLE table=test-table [id=758d4d8eaf494f5c91aa954f927f1097]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:04:43.099733 11163 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36e78fa7c12d4ad3920e31c608b0cc83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:43.100355 11539 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:43.094542 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37490 (local address 127.2.197.195:37495)
0429 14:04:43.094923 (+ 381us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:43.094931 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:43.094942 (+ 11us) server_negotiation.cc:408] Connection header received
0429 14:04:43.094980 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:43.094982 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:43.094998 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:43.095070 (+ 72us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:04:43.096854 (+ 1784us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.097660 (+ 806us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:43.099682 (+ 2022us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.100165 (+ 483us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:43.100206 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:43.100235 (+ 29us) server_negotiation.cc:300] Negotiation successful
0429 14:04:43.100280 (+ 45us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":118,"threads_started":1}
I20260429 14:04:43.100821 11447 tablet_service.cc:1511] Processing CreateTablet for tablet 36e78fa7c12d4ad3920e31c608b0cc83 (DEFAULT_TABLE table=test-table [id=758d4d8eaf494f5c91aa954f927f1097]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:04:43.101074 11447 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36e78fa7c12d4ad3920e31c608b0cc83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:43.102308 11541 tablet_bootstrap.cc:492] T 36e78fa7c12d4ad3920e31c608b0cc83 P 49573166d38746089e79505c86c6e93a: Bootstrap starting.
I20260429 14:04:43.102450 11542 tablet_bootstrap.cc:492] T 36e78fa7c12d4ad3920e31c608b0cc83 P 934c8c1fa4f64ba7941168f5ab504ce7: Bootstrap starting.
I20260429 14:04:43.102923 11541 tablet_bootstrap.cc:654] T 36e78fa7c12d4ad3920e31c608b0cc83 P 49573166d38746089e79505c86c6e93a: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:43.103051 11542 tablet_bootstrap.cc:654] T 36e78fa7c12d4ad3920e31c608b0cc83 P 934c8c1fa4f64ba7941168f5ab504ce7: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:43.103503 11541 log.cc:826] T 36e78fa7c12d4ad3920e31c608b0cc83 P 49573166d38746089e79505c86c6e93a: Log is configured to *not* fsync() on all Append() calls
I20260429 14:04:43.103596 11542 log.cc:826] T 36e78fa7c12d4ad3920e31c608b0cc83 P 934c8c1fa4f64ba7941168f5ab504ce7: Log is configured to *not* fsync() on all Append() calls
I20260429 14:04:43.104152 11544 tablet_bootstrap.cc:492] T 36e78fa7c12d4ad3920e31c608b0cc83 P f49e2dd467f042f08da4345b01c63e9b: Bootstrap starting.
I20260429 14:04:43.104285 11541 tablet_bootstrap.cc:492] T 36e78fa7c12d4ad3920e31c608b0cc83 P 49573166d38746089e79505c86c6e93a: No bootstrap required, opened a new log
I20260429 14:04:43.104310 11542 tablet_bootstrap.cc:492] T 36e78fa7c12d4ad3920e31c608b0cc83 P 934c8c1fa4f64ba7941168f5ab504ce7: No bootstrap required, opened a new log
I20260429 14:04:43.104351 11541 ts_tablet_manager.cc:1403] T 36e78fa7c12d4ad3920e31c608b0cc83 P 49573166d38746089e79505c86c6e93a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260429 14:04:43.104367 11542 ts_tablet_manager.cc:1403] T 36e78fa7c12d4ad3920e31c608b0cc83 P 934c8c1fa4f64ba7941168f5ab504ce7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260429 14:04:43.104725 11544 tablet_bootstrap.cc:654] T 36e78fa7c12d4ad3920e31c608b0cc83 P f49e2dd467f042f08da4345b01c63e9b: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:43.105355 11544 log.cc:826] T 36e78fa7c12d4ad3920e31c608b0cc83 P f49e2dd467f042f08da4345b01c63e9b: Log is configured to *not* fsync() on all Append() calls
I20260429 14:04:43.105650 11541 raft_consensus.cc:359] T 36e78fa7c12d4ad3920e31c608b0cc83 P 49573166d38746089e79505c86c6e93a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:43.105821 11541 raft_consensus.cc:385] T 36e78fa7c12d4ad3920e31c608b0cc83 P 49573166d38746089e79505c86c6e93a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:43.105863 11541 raft_consensus.cc:740] T 36e78fa7c12d4ad3920e31c608b0cc83 P 49573166d38746089e79505c86c6e93a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49573166d38746089e79505c86c6e93a, State: Initialized, Role: FOLLOWER
I20260429 14:04:43.105973 11541 consensus_queue.cc:260] T 36e78fa7c12d4ad3920e31c608b0cc83 P 49573166d38746089e79505c86c6e93a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:43.106248 11541 ts_tablet_manager.cc:1434] T 36e78fa7c12d4ad3920e31c608b0cc83 P 49573166d38746089e79505c86c6e93a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260429 14:04:43.106251 11544 tablet_bootstrap.cc:492] T 36e78fa7c12d4ad3920e31c608b0cc83 P f49e2dd467f042f08da4345b01c63e9b: No bootstrap required, opened a new log
I20260429 14:04:43.106195 11542 raft_consensus.cc:359] T 36e78fa7c12d4ad3920e31c608b0cc83 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:43.106396 11229 heartbeater.cc:499] Master 127.2.197.254:42715 was elected leader, sending a full tablet report...
I20260429 14:04:43.106403 11542 raft_consensus.cc:385] T 36e78fa7c12d4ad3920e31c608b0cc83 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:43.106442 11542 raft_consensus.cc:740] T 36e78fa7c12d4ad3920e31c608b0cc83 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 934c8c1fa4f64ba7941168f5ab504ce7, State: Initialized, Role: FOLLOWER
I20260429 14:04:43.106361 11544 ts_tablet_manager.cc:1403] T 36e78fa7c12d4ad3920e31c608b0cc83 P f49e2dd467f042f08da4345b01c63e9b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260429 14:04:43.106539 11542 consensus_queue.cc:260] T 36e78fa7c12d4ad3920e31c608b0cc83 P 934c8c1fa4f64ba7941168f5ab504ce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:43.106839 11371 heartbeater.cc:499] Master 127.2.197.254:42715 was elected leader, sending a full tablet report...
I20260429 14:04:43.107069 11542 ts_tablet_manager.cc:1434] T 36e78fa7c12d4ad3920e31c608b0cc83 P 934c8c1fa4f64ba7941168f5ab504ce7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260429 14:04:43.108244 11544 raft_consensus.cc:359] T 36e78fa7c12d4ad3920e31c608b0cc83 P f49e2dd467f042f08da4345b01c63e9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:43.108372 11544 raft_consensus.cc:385] T 36e78fa7c12d4ad3920e31c608b0cc83 P f49e2dd467f042f08da4345b01c63e9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:43.108415 11544 raft_consensus.cc:740] T 36e78fa7c12d4ad3920e31c608b0cc83 P f49e2dd467f042f08da4345b01c63e9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f49e2dd467f042f08da4345b01c63e9b, State: Initialized, Role: FOLLOWER
I20260429 14:04:43.108506 11544 consensus_queue.cc:260] T 36e78fa7c12d4ad3920e31c608b0cc83 P f49e2dd467f042f08da4345b01c63e9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:43.108747 11544 ts_tablet_manager.cc:1434] T 36e78fa7c12d4ad3920e31c608b0cc83 P f49e2dd467f042f08da4345b01c63e9b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260429 14:04:43.108807 11513 heartbeater.cc:499] Master 127.2.197.254:42715 was elected leader, sending a full tablet report...
W20260429 14:04:43.113876 11372 tablet.cc:2404] T 36e78fa7c12d4ad3920e31c608b0cc83 P 934c8c1fa4f64ba7941168f5ab504ce7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260429 14:04:43.181586 11230 tablet.cc:2404] T 36e78fa7c12d4ad3920e31c608b0cc83 P 49573166d38746089e79505c86c6e93a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 14:04:43.231165 11549 raft_consensus.cc:493] T 36e78fa7c12d4ad3920e31c608b0cc83 P f49e2dd467f042f08da4345b01c63e9b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:04:43.231287 11549 raft_consensus.cc:515] T 36e78fa7c12d4ad3920e31c608b0cc83 P f49e2dd467f042f08da4345b01c63e9b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:43.231633 11549 leader_election.cc:290] T 36e78fa7c12d4ad3920e31c608b0cc83 P f49e2dd467f042f08da4345b01c63e9b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49573166d38746089e79505c86c6e93a (127.2.197.193:42967), 934c8c1fa4f64ba7941168f5ab504ce7 (127.2.197.194:37361)
I20260429 14:04:43.233839 11406 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:43.231847 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:37361 (local address 127.2.197.195:46725)
0429 14:04:43.231903 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:43.231916 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:43.231963 (+ 47us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:43.232127 (+ 164us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:43.232128 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:43.232130 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:04:43.232313 (+ 183us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:43.232315 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.233025 (+ 710us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:43.233026 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:43.233594 (+ 568us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:43.233596 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.233692 (+ 96us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:43.233697 (+ 5us) client_negotiation.cc:770] Sending connection context
0429 14:04:43.233729 (+ 32us) client_negotiation.cc:241] Negotiation successful
0429 14:04:43.233769 (+ 40us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260429 14:04:43.234227 11550 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:43.231866 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:42967 (local address 127.2.197.195:33171)
0429 14:04:43.232039 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:43.232055 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:43.232141 (+ 86us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:43.232290 (+ 149us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:43.232291 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:43.232293 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:04:43.232503 (+ 210us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:43.232505 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.233208 (+ 703us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:43.233209 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:43.233984 (+ 775us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:43.233987 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.234088 (+ 101us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:43.234091 (+ 3us) client_negotiation.cc:770] Sending connection context
0429 14:04:43.234123 (+ 32us) client_negotiation.cc:241] Negotiation successful
0429 14:04:43.234159 (+ 36us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":122,"threads_started":1}
I20260429 14:04:43.234333 11540 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:43.231924 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:46725 (local address 127.2.197.194:37361)
0429 14:04:43.231964 (+ 40us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:43.231966 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:43.231976 (+ 10us) server_negotiation.cc:408] Connection header received
0429 14:04:43.232034 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:43.232036 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:43.232051 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:43.232122 (+ 71us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:04:43.232402 (+ 280us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.232931 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:43.233690 (+ 759us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.234191 (+ 501us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:43.234200 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:43.234222 (+ 22us) server_negotiation.cc:300] Negotiation successful
0429 14:04:43.234254 (+ 32us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260429 14:04:43.234649 11325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e78fa7c12d4ad3920e31c608b0cc83" candidate_uuid: "f49e2dd467f042f08da4345b01c63e9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" is_pre_election: true
I20260429 14:04:43.234709 11536 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:43.231901 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:33171 (local address 127.2.197.193:42967)
0429 14:04:43.231960 (+ 59us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:43.231962 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:43.232125 (+ 163us) server_negotiation.cc:408] Connection header received
0429 14:04:43.232200 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:43.232202 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:43.232212 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:43.232287 (+ 75us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:04:43.232649 (+ 362us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.233115 (+ 466us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:43.234123 (+ 1008us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.234610 (+ 487us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:43.234618 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:43.234639 (+ 21us) server_negotiation.cc:300] Negotiation successful
0429 14:04:43.234663 (+ 24us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260429 14:04:43.234833 11325 raft_consensus.cc:2468] T 36e78fa7c12d4ad3920e31c608b0cc83 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f49e2dd467f042f08da4345b01c63e9b in term 0.
I20260429 14:04:43.235059 11183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e78fa7c12d4ad3920e31c608b0cc83" candidate_uuid: "f49e2dd467f042f08da4345b01c63e9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49573166d38746089e79505c86c6e93a" is_pre_election: true
I20260429 14:04:43.235217 11183 raft_consensus.cc:2468] T 36e78fa7c12d4ad3920e31c608b0cc83 P 49573166d38746089e79505c86c6e93a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f49e2dd467f042f08da4345b01c63e9b in term 0.
I20260429 14:04:43.235299 11399 leader_election.cc:304] T 36e78fa7c12d4ad3920e31c608b0cc83 P f49e2dd467f042f08da4345b01c63e9b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 934c8c1fa4f64ba7941168f5ab504ce7, f49e2dd467f042f08da4345b01c63e9b; no voters:
I20260429 14:04:43.235435 11549 raft_consensus.cc:2804] T 36e78fa7c12d4ad3920e31c608b0cc83 P f49e2dd467f042f08da4345b01c63e9b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 14:04:43.235532 11549 raft_consensus.cc:493] T 36e78fa7c12d4ad3920e31c608b0cc83 P f49e2dd467f042f08da4345b01c63e9b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 14:04:43.235553 11549 raft_consensus.cc:3060] T 36e78fa7c12d4ad3920e31c608b0cc83 P f49e2dd467f042f08da4345b01c63e9b [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:43.236274 11549 raft_consensus.cc:515] T 36e78fa7c12d4ad3920e31c608b0cc83 P f49e2dd467f042f08da4345b01c63e9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:43.236459 11549 leader_election.cc:290] T 36e78fa7c12d4ad3920e31c608b0cc83 P f49e2dd467f042f08da4345b01c63e9b [CANDIDATE]: Term 1 election: Requested vote from peers 49573166d38746089e79505c86c6e93a (127.2.197.193:42967), 934c8c1fa4f64ba7941168f5ab504ce7 (127.2.197.194:37361)
I20260429 14:04:43.236579 11183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e78fa7c12d4ad3920e31c608b0cc83" candidate_uuid: "f49e2dd467f042f08da4345b01c63e9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49573166d38746089e79505c86c6e93a"
I20260429 14:04:43.236668 11183 raft_consensus.cc:3060] T 36e78fa7c12d4ad3920e31c608b0cc83 P 49573166d38746089e79505c86c6e93a [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:43.236670 11325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e78fa7c12d4ad3920e31c608b0cc83" candidate_uuid: "f49e2dd467f042f08da4345b01c63e9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "934c8c1fa4f64ba7941168f5ab504ce7"
I20260429 14:04:43.236735 11325 raft_consensus.cc:3060] T 36e78fa7c12d4ad3920e31c608b0cc83 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:43.237598 11183 raft_consensus.cc:2468] T 36e78fa7c12d4ad3920e31c608b0cc83 P 49573166d38746089e79505c86c6e93a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f49e2dd467f042f08da4345b01c63e9b in term 1.
I20260429 14:04:43.237598 11325 raft_consensus.cc:2468] T 36e78fa7c12d4ad3920e31c608b0cc83 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f49e2dd467f042f08da4345b01c63e9b in term 1.
I20260429 14:04:43.237834 11400 leader_election.cc:304] T 36e78fa7c12d4ad3920e31c608b0cc83 P f49e2dd467f042f08da4345b01c63e9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49573166d38746089e79505c86c6e93a, f49e2dd467f042f08da4345b01c63e9b; no voters:
I20260429 14:04:43.238001 11549 raft_consensus.cc:2804] T 36e78fa7c12d4ad3920e31c608b0cc83 P f49e2dd467f042f08da4345b01c63e9b [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:04:43.238199 11549 raft_consensus.cc:697] T 36e78fa7c12d4ad3920e31c608b0cc83 P f49e2dd467f042f08da4345b01c63e9b [term 1 LEADER]: Becoming Leader. State: Replica: f49e2dd467f042f08da4345b01c63e9b, State: Running, Role: LEADER
I20260429 14:04:43.238315 11549 consensus_queue.cc:237] T 36e78fa7c12d4ad3920e31c608b0cc83 P f49e2dd467f042f08da4345b01c63e9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:43.239288 11024 catalog_manager.cc:5671] T 36e78fa7c12d4ad3920e31c608b0cc83 P f49e2dd467f042f08da4345b01c63e9b reported cstate change: term changed from 0 to 1, leader changed from <none> to f49e2dd467f042f08da4345b01c63e9b (127.2.197.195). New cstate: current_term: 1 leader_uuid: "f49e2dd467f042f08da4345b01c63e9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } health_report { overall_health: HEALTHY } } }
I20260429 14:04:43.279997 11539 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:43.277630 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37506 (local address 127.2.197.195:37495)
0429 14:04:43.277810 (+ 180us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:43.277812 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:43.277857 (+ 45us) server_negotiation.cc:408] Connection header received
0429 14:04:43.277961 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:43.277962 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:43.277977 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:43.278048 (+ 71us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:04:43.278360 (+ 312us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.278856 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:43.279455 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.279618 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:43.279648 (+ 30us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:04:43.279733 (+ 85us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:04:43.279814 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:43.279885 (+ 71us) server_negotiation.cc:300] Negotiation successful
0429 14:04:43.279909 (+ 24us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260429 14:04:43.281895 11183 raft_consensus.cc:1275] T 36e78fa7c12d4ad3920e31c608b0cc83 P 49573166d38746089e79505c86c6e93a [term 1 FOLLOWER]: Refusing update from remote peer f49e2dd467f042f08da4345b01c63e9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:04:43.281942 11325 raft_consensus.cc:1275] T 36e78fa7c12d4ad3920e31c608b0cc83 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 1 FOLLOWER]: Refusing update from remote peer f49e2dd467f042f08da4345b01c63e9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:04:43.282258 11551 consensus_queue.cc:1048] T 36e78fa7c12d4ad3920e31c608b0cc83 P f49e2dd467f042f08da4345b01c63e9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:04:43.282480 11549 consensus_queue.cc:1048] T 36e78fa7c12d4ad3920e31c608b0cc83 P f49e2dd467f042f08da4345b01c63e9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:04:43.287324 11024 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:04:43.287454 11024 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:04:43.288342 11024 catalog_manager.cc:5958] T 00000000000000000000000000000000 P bb7ccf4eb15e4d9e84fe3634fa4b154b: Sending DeleteTablet for 3 replicas of tablet 36e78fa7c12d4ad3920e31c608b0cc83
I20260429 14:04:43.288646 11163 tablet_service.cc:1558] Processing DeleteTablet for tablet 36e78fa7c12d4ad3920e31c608b0cc83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:04:43 UTC) from {username='oryx', principal='oryx/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:39604
I20260429 14:04:43.288672 11447 tablet_service.cc:1558] Processing DeleteTablet for tablet 36e78fa7c12d4ad3920e31c608b0cc83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:04:43 UTC) from {username='oryx', principal='oryx/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:37490
I20260429 14:04:43.288748 11305 tablet_service.cc:1558] Processing DeleteTablet for tablet 36e78fa7c12d4ad3920e31c608b0cc83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:04:43 UTC) from {username='oryx', principal='oryx/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:38694
I20260429 14:04:43.288856 11564 tablet_replica.cc:333] T 36e78fa7c12d4ad3920e31c608b0cc83 P 49573166d38746089e79505c86c6e93a: stopping tablet replica
I20260429 14:04:43.288933 11564 raft_consensus.cc:2243] T 36e78fa7c12d4ad3920e31c608b0cc83 P 49573166d38746089e79505c86c6e93a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:04:43.288991 11566 tablet_replica.cc:333] T 36e78fa7c12d4ad3920e31c608b0cc83 P 934c8c1fa4f64ba7941168f5ab504ce7: stopping tablet replica
I20260429 14:04:43.289021 11564 raft_consensus.cc:2272] T 36e78fa7c12d4ad3920e31c608b0cc83 P 49573166d38746089e79505c86c6e93a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:04:43.289067 11566 raft_consensus.cc:2243] T 36e78fa7c12d4ad3920e31c608b0cc83 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:04:43.289160 11566 raft_consensus.cc:2272] T 36e78fa7c12d4ad3920e31c608b0cc83 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:04:43.289343 11024 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260429 14:04:43.289469 11564 ts_tablet_manager.cc:1916] T 36e78fa7c12d4ad3920e31c608b0cc83 P 49573166d38746089e79505c86c6e93a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260429 14:04:43.289501 11024 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 14:04:43.289666 11565 tablet_replica.cc:333] T 36e78fa7c12d4ad3920e31c608b0cc83 P f49e2dd467f042f08da4345b01c63e9b: stopping tablet replica
I20260429 14:04:43.289805 11565 raft_consensus.cc:2243] T 36e78fa7c12d4ad3920e31c608b0cc83 P f49e2dd467f042f08da4345b01c63e9b [term 1 LEADER]: Raft consensus shutting down.
I20260429 14:04:43.289966 11565 raft_consensus.cc:2272] T 36e78fa7c12d4ad3920e31c608b0cc83 P f49e2dd467f042f08da4345b01c63e9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:04:43.290473 11565 ts_tablet_manager.cc:1916] T 36e78fa7c12d4ad3920e31c608b0cc83 P f49e2dd467f042f08da4345b01c63e9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:04:43.290899 11163 tablet_service.cc:1511] Processing CreateTablet for tablet 2473765fd7df4b3d93d22e582f10b126 (DEFAULT_TABLE table=test-table [id=823a76f281ac4203aa3ac9c65bbd069b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:04:43.291018 11163 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2473765fd7df4b3d93d22e582f10b126. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:43.291153 11564 ts_tablet_manager.cc:1929] T 36e78fa7c12d4ad3920e31c608b0cc83 P 49573166d38746089e79505c86c6e93a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 14:04:43.291234 11564 log.cc:1199] T 36e78fa7c12d4ad3920e31c608b0cc83 P 49573166d38746089e79505c86c6e93a: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-0/wal/wals/36e78fa7c12d4ad3920e31c608b0cc83
I20260429 14:04:43.291517 11564 ts_tablet_manager.cc:1950] T 36e78fa7c12d4ad3920e31c608b0cc83 P 49573166d38746089e79505c86c6e93a: Deleting consensus metadata
I20260429 14:04:43.292255 11010 catalog_manager.cc:5002] TS 49573166d38746089e79505c86c6e93a (127.2.197.193:42967): tablet 36e78fa7c12d4ad3920e31c608b0cc83 (table test-table [id=758d4d8eaf494f5c91aa954f927f1097]) successfully deleted
I20260429 14:04:43.292349 11447 tablet_service.cc:1511] Processing CreateTablet for tablet 2473765fd7df4b3d93d22e582f10b126 (DEFAULT_TABLE table=test-table [id=823a76f281ac4203aa3ac9c65bbd069b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:04:43.292384 11541 tablet_bootstrap.cc:492] T 2473765fd7df4b3d93d22e582f10b126 P 49573166d38746089e79505c86c6e93a: Bootstrap starting.
I20260429 14:04:43.292582 11565 ts_tablet_manager.cc:1929] T 36e78fa7c12d4ad3920e31c608b0cc83 P f49e2dd467f042f08da4345b01c63e9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 14:04:43.292694 11565 log.cc:1199] T 36e78fa7c12d4ad3920e31c608b0cc83 P f49e2dd467f042f08da4345b01c63e9b: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-2/wal/wals/36e78fa7c12d4ad3920e31c608b0cc83
I20260429 14:04:43.292804 11541 tablet_bootstrap.cc:654] T 2473765fd7df4b3d93d22e582f10b126 P 49573166d38746089e79505c86c6e93a: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:43.292832 11305 tablet_service.cc:1511] Processing CreateTablet for tablet 2473765fd7df4b3d93d22e582f10b126 (DEFAULT_TABLE table=test-table [id=823a76f281ac4203aa3ac9c65bbd069b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:04:43.292941 11305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2473765fd7df4b3d93d22e582f10b126. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:43.293143 11565 ts_tablet_manager.cc:1950] T 36e78fa7c12d4ad3920e31c608b0cc83 P f49e2dd467f042f08da4345b01c63e9b: Deleting consensus metadata
I20260429 14:04:43.293560 11541 tablet_bootstrap.cc:492] T 2473765fd7df4b3d93d22e582f10b126 P 49573166d38746089e79505c86c6e93a: No bootstrap required, opened a new log
I20260429 14:04:43.293615 11541 ts_tablet_manager.cc:1403] T 2473765fd7df4b3d93d22e582f10b126 P 49573166d38746089e79505c86c6e93a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:43.293783 11541 raft_consensus.cc:359] T 2473765fd7df4b3d93d22e582f10b126 P 49573166d38746089e79505c86c6e93a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:43.293859 11541 raft_consensus.cc:385] T 2473765fd7df4b3d93d22e582f10b126 P 49573166d38746089e79505c86c6e93a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:43.293880 11541 raft_consensus.cc:740] T 2473765fd7df4b3d93d22e582f10b126 P 49573166d38746089e79505c86c6e93a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49573166d38746089e79505c86c6e93a, State: Initialized, Role: FOLLOWER
I20260429 14:04:43.293924 11541 consensus_queue.cc:260] T 2473765fd7df4b3d93d22e582f10b126 P 49573166d38746089e79505c86c6e93a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:43.294023 11541 ts_tablet_manager.cc:1434] T 2473765fd7df4b3d93d22e582f10b126 P 49573166d38746089e79505c86c6e93a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260429 14:04:43.294320 11542 tablet_bootstrap.cc:492] T 2473765fd7df4b3d93d22e582f10b126 P 934c8c1fa4f64ba7941168f5ab504ce7: Bootstrap starting.
I20260429 14:04:43.294773 11542 tablet_bootstrap.cc:654] T 2473765fd7df4b3d93d22e582f10b126 P 934c8c1fa4f64ba7941168f5ab504ce7: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:43.295303 11447 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2473765fd7df4b3d93d22e582f10b126. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:43.296350 11544 tablet_bootstrap.cc:492] T 2473765fd7df4b3d93d22e582f10b126 P f49e2dd467f042f08da4345b01c63e9b: Bootstrap starting.
I20260429 14:04:43.296686 11542 tablet_bootstrap.cc:492] T 2473765fd7df4b3d93d22e582f10b126 P 934c8c1fa4f64ba7941168f5ab504ce7: No bootstrap required, opened a new log
I20260429 14:04:43.296758 11542 ts_tablet_manager.cc:1403] T 2473765fd7df4b3d93d22e582f10b126 P 934c8c1fa4f64ba7941168f5ab504ce7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260429 14:04:43.296898 11544 tablet_bootstrap.cc:654] T 2473765fd7df4b3d93d22e582f10b126 P f49e2dd467f042f08da4345b01c63e9b: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:43.297093 11542 raft_consensus.cc:359] T 2473765fd7df4b3d93d22e582f10b126 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:43.297197 11542 raft_consensus.cc:385] T 2473765fd7df4b3d93d22e582f10b126 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:43.297220 11542 raft_consensus.cc:740] T 2473765fd7df4b3d93d22e582f10b126 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 934c8c1fa4f64ba7941168f5ab504ce7, State: Initialized, Role: FOLLOWER
I20260429 14:04:43.297263 11542 consensus_queue.cc:260] T 2473765fd7df4b3d93d22e582f10b126 P 934c8c1fa4f64ba7941168f5ab504ce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:43.297370 11542 ts_tablet_manager.cc:1434] T 2473765fd7df4b3d93d22e582f10b126 P 934c8c1fa4f64ba7941168f5ab504ce7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:43.297808 11544 tablet_bootstrap.cc:492] T 2473765fd7df4b3d93d22e582f10b126 P f49e2dd467f042f08da4345b01c63e9b: No bootstrap required, opened a new log
I20260429 14:04:43.297858 11544 ts_tablet_manager.cc:1403] T 2473765fd7df4b3d93d22e582f10b126 P f49e2dd467f042f08da4345b01c63e9b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260429 14:04:43.298012 11544 raft_consensus.cc:359] T 2473765fd7df4b3d93d22e582f10b126 P f49e2dd467f042f08da4345b01c63e9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:43.298095 11544 raft_consensus.cc:385] T 2473765fd7df4b3d93d22e582f10b126 P f49e2dd467f042f08da4345b01c63e9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:43.298120 11544 raft_consensus.cc:740] T 2473765fd7df4b3d93d22e582f10b126 P f49e2dd467f042f08da4345b01c63e9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f49e2dd467f042f08da4345b01c63e9b, State: Initialized, Role: FOLLOWER
I20260429 14:04:43.298115 11008 catalog_manager.cc:5002] TS f49e2dd467f042f08da4345b01c63e9b (127.2.197.195:37495): tablet 36e78fa7c12d4ad3920e31c608b0cc83 (table test-table [id=758d4d8eaf494f5c91aa954f927f1097]) successfully deleted
I20260429 14:04:43.298174 11544 consensus_queue.cc:260] T 2473765fd7df4b3d93d22e582f10b126 P f49e2dd467f042f08da4345b01c63e9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:43.298300 11544 ts_tablet_manager.cc:1434] T 2473765fd7df4b3d93d22e582f10b126 P f49e2dd467f042f08da4345b01c63e9b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260429 14:04:43.298588 11566 ts_tablet_manager.cc:1916] T 36e78fa7c12d4ad3920e31c608b0cc83 P 934c8c1fa4f64ba7941168f5ab504ce7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:04:43.300017 11566 ts_tablet_manager.cc:1929] T 36e78fa7c12d4ad3920e31c608b0cc83 P 934c8c1fa4f64ba7941168f5ab504ce7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 14:04:43.300086 11566 log.cc:1199] T 36e78fa7c12d4ad3920e31c608b0cc83 P 934c8c1fa4f64ba7941168f5ab504ce7: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-1/wal/wals/36e78fa7c12d4ad3920e31c608b0cc83
I20260429 14:04:43.300320 11566 ts_tablet_manager.cc:1950] T 36e78fa7c12d4ad3920e31c608b0cc83 P 934c8c1fa4f64ba7941168f5ab504ce7: Deleting consensus metadata
I20260429 14:04:43.300690 11009 catalog_manager.cc:5002] TS 934c8c1fa4f64ba7941168f5ab504ce7 (127.2.197.194:37361): tablet 36e78fa7c12d4ad3920e31c608b0cc83 (table test-table [id=758d4d8eaf494f5c91aa954f927f1097]) successfully deleted
I20260429 14:04:43.393512 11547 raft_consensus.cc:493] T 2473765fd7df4b3d93d22e582f10b126 P 49573166d38746089e79505c86c6e93a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:04:43.393639 11547 raft_consensus.cc:515] T 2473765fd7df4b3d93d22e582f10b126 P 49573166d38746089e79505c86c6e93a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:43.393999 11547 leader_election.cc:290] T 2473765fd7df4b3d93d22e582f10b126 P 49573166d38746089e79505c86c6e93a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 934c8c1fa4f64ba7941168f5ab504ce7 (127.2.197.194:37361), f49e2dd467f042f08da4345b01c63e9b (127.2.197.195:37495)
I20260429 14:04:43.397094 11571 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:43.394228 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:37495 (local address 127.2.197.193:42231)
0429 14:04:43.394512 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:43.394523 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:43.394592 (+ 69us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:43.394725 (+ 133us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:43.394726 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:43.394727 (+ 1us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:04:43.395000 (+ 273us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:43.395004 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.395969 (+ 965us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:43.395971 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:43.396800 (+ 829us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:43.396803 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.396924 (+ 121us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:43.396929 (+ 5us) client_negotiation.cc:770] Sending connection context
0429 14:04:43.396963 (+ 34us) client_negotiation.cc:241] Negotiation successful
0429 14:04:43.397003 (+ 40us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":117,"threads_started":1}
I20260429 14:04:43.397531 11539 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:43.394323 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:42231 (local address 127.2.197.195:37495)
0429 14:04:43.394374 (+ 51us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:43.394376 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:43.394592 (+ 216us) server_negotiation.cc:408] Connection header received
0429 14:04:43.394649 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:43.394651 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:43.394661 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:43.394712 (+ 51us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:04:43.395116 (+ 404us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.395859 (+ 743us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:43.396919 (+ 1060us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.397398 (+ 479us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:43.397407 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:43.397430 (+ 23us) server_negotiation.cc:300] Negotiation successful
0429 14:04:43.397464 (+ 34us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260429 14:04:43.397786 11540 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:43.394428 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:53123 (local address 127.2.197.194:37361)
0429 14:04:43.394490 (+ 62us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:43.394493 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:43.394504 (+ 11us) server_negotiation.cc:408] Connection header received
0429 14:04:43.394592 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:43.394593 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:43.394604 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:43.394654 (+ 50us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:04:43.395446 (+ 792us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.395930 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:43.397046 (+ 1116us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.397536 (+ 490us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:43.397544 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:43.397677 (+ 133us) server_negotiation.cc:300] Negotiation successful
0429 14:04:43.397720 (+ 43us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260429 14:04:43.397819 11467 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2473765fd7df4b3d93d22e582f10b126" candidate_uuid: "49573166d38746089e79505c86c6e93a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f49e2dd467f042f08da4345b01c63e9b" is_pre_election: true
I20260429 14:04:43.397959 11467 raft_consensus.cc:2468] T 2473765fd7df4b3d93d22e582f10b126 P f49e2dd467f042f08da4345b01c63e9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49573166d38746089e79505c86c6e93a in term 0.
I20260429 14:04:43.398041 11570 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:43.394185 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:37361 (local address 127.2.197.193:53123)
0429 14:04:43.394417 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:43.394429 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:43.394511 (+ 82us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:43.394675 (+ 164us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:43.394676 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:43.394678 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:04:43.395359 (+ 681us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:43.395362 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.396032 (+ 670us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:43.396034 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:43.396931 (+ 897us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:43.396934 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.397581 (+ 647us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:43.397585 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 14:04:43.397919 (+ 334us) client_negotiation.cc:241] Negotiation successful
0429 14:04:43.397964 (+ 45us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":64,"threads_started":1}
I20260429 14:04:43.398155 11114 leader_election.cc:304] T 2473765fd7df4b3d93d22e582f10b126 P 49573166d38746089e79505c86c6e93a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49573166d38746089e79505c86c6e93a, f49e2dd467f042f08da4345b01c63e9b; no voters:
I20260429 14:04:43.398283 11547 raft_consensus.cc:2804] T 2473765fd7df4b3d93d22e582f10b126 P 49573166d38746089e79505c86c6e93a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 14:04:43.398263 11325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2473765fd7df4b3d93d22e582f10b126" candidate_uuid: "49573166d38746089e79505c86c6e93a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" is_pre_election: true
I20260429 14:04:43.398336 11547 raft_consensus.cc:493] T 2473765fd7df4b3d93d22e582f10b126 P 49573166d38746089e79505c86c6e93a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 14:04:43.398356 11547 raft_consensus.cc:3060] T 2473765fd7df4b3d93d22e582f10b126 P 49573166d38746089e79505c86c6e93a [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:43.398365 11325 raft_consensus.cc:2468] T 2473765fd7df4b3d93d22e582f10b126 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49573166d38746089e79505c86c6e93a in term 0.
I20260429 14:04:43.399183 11547 raft_consensus.cc:515] T 2473765fd7df4b3d93d22e582f10b126 P 49573166d38746089e79505c86c6e93a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:43.399354 11547 leader_election.cc:290] T 2473765fd7df4b3d93d22e582f10b126 P 49573166d38746089e79505c86c6e93a [CANDIDATE]: Term 1 election: Requested vote from peers 934c8c1fa4f64ba7941168f5ab504ce7 (127.2.197.194:37361), f49e2dd467f042f08da4345b01c63e9b (127.2.197.195:37495)
I20260429 14:04:43.399526 11325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2473765fd7df4b3d93d22e582f10b126" candidate_uuid: "49573166d38746089e79505c86c6e93a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "934c8c1fa4f64ba7941168f5ab504ce7"
I20260429 14:04:43.399561 11467 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2473765fd7df4b3d93d22e582f10b126" candidate_uuid: "49573166d38746089e79505c86c6e93a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f49e2dd467f042f08da4345b01c63e9b"
I20260429 14:04:43.399632 11325 raft_consensus.cc:3060] T 2473765fd7df4b3d93d22e582f10b126 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:43.399645 11467 raft_consensus.cc:3060] T 2473765fd7df4b3d93d22e582f10b126 P f49e2dd467f042f08da4345b01c63e9b [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:43.400213 11325 raft_consensus.cc:2468] T 2473765fd7df4b3d93d22e582f10b126 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49573166d38746089e79505c86c6e93a in term 1.
I20260429 14:04:43.400421 11115 leader_election.cc:304] T 2473765fd7df4b3d93d22e582f10b126 P 49573166d38746089e79505c86c6e93a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49573166d38746089e79505c86c6e93a, 934c8c1fa4f64ba7941168f5ab504ce7; no voters:
I20260429 14:04:43.400462 11467 raft_consensus.cc:2468] T 2473765fd7df4b3d93d22e582f10b126 P f49e2dd467f042f08da4345b01c63e9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49573166d38746089e79505c86c6e93a in term 1.
I20260429 14:04:43.400532 11547 raft_consensus.cc:2804] T 2473765fd7df4b3d93d22e582f10b126 P 49573166d38746089e79505c86c6e93a [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:04:43.400696 11547 raft_consensus.cc:697] T 2473765fd7df4b3d93d22e582f10b126 P 49573166d38746089e79505c86c6e93a [term 1 LEADER]: Becoming Leader. State: Replica: 49573166d38746089e79505c86c6e93a, State: Running, Role: LEADER
I20260429 14:04:43.400772 11547 consensus_queue.cc:237] T 2473765fd7df4b3d93d22e582f10b126 P 49573166d38746089e79505c86c6e93a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:43.401508 11024 catalog_manager.cc:5671] T 2473765fd7df4b3d93d22e582f10b126 P 49573166d38746089e79505c86c6e93a reported cstate change: term changed from 0 to 1, leader changed from <none> to 49573166d38746089e79505c86c6e93a (127.2.197.193). New cstate: current_term: 1 leader_uuid: "49573166d38746089e79505c86c6e93a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } health_report { overall_health: UNKNOWN } } }
Apr 29 14:04:43 dist-test-slave-8n77 krb5kdc[10976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471482, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.254@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM
I20260429 14:04:43.438107 11579 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:43.431153 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:42715 (local address 127.0.0.1:57836)
0429 14:04:43.431342 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:43.431360 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:43.431439 (+ 79us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:43.431622 (+ 183us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:43.431623 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:43.432001 (+ 378us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:43.432204 (+ 203us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:43.432206 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.432844 (+ 638us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:43.432845 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:43.433209 (+ 364us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:43.433211 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.433291 (+ 80us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:43.433889 (+ 598us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:43.433895 (+ 6us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:43.435511 (+ 1616us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:43.437225 (+ 1714us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:43.437230 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:43.437233 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:43.437429 (+ 196us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:43.437594 (+ 165us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:43.437595 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:43.437595 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:43.437627 (+ 32us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:43.437831 (+ 204us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:43.437836 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:43.437878 (+ 42us) client_negotiation.cc:770] Sending connection context
0429 14:04:43.437932 (+ 54us) client_negotiation.cc:241] Negotiation successful
0429 14:04:43.438033 (+ 101us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":100,"threads_started":1}
I20260429 14:04:43.438215 11073 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:43.431213 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57836 (local address 127.2.197.254:42715)
0429 14:04:43.431288 (+ 75us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:43.431291 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:43.431427 (+ 136us) server_negotiation.cc:408] Connection header received
0429 14:04:43.431510 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:43.431511 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:43.431523 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:43.431569 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:43.432290 (+ 721us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.432756 (+ 466us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:43.433299 (+ 543us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.433477 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:43.435644 (+ 2167us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:43.435650 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:43.435650 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:43.435658 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:43.437137 (+ 1479us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:43.437496 (+ 359us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:43.437497 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:43.437497 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:43.437517 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:43.437690 (+ 173us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:43.437691 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:43.437691 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:43.437776 (+ 85us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:43.437829 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:43.437960 (+ 131us) server_negotiation.cc:300] Negotiation successful
0429 14:04:43.438093 (+ 133us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":45}
I20260429 14:04:43.440172 11024 catalog_manager.cc:2257] Servicing CreateTable request from {username='oryx', principal='oryx/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:57836:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260429 14:04:43.440349 11024 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 14:04:43.441860 11447 tablet_service.cc:1511] Processing CreateTablet for tablet 3efb00c93ab44cf4864b468153d75028 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d572e29aab024f6cb916fbe5fab49b51]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260429 14:04:43.441879 11163 tablet_service.cc:1511] Processing CreateTablet for tablet 3efb00c93ab44cf4864b468153d75028 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d572e29aab024f6cb916fbe5fab49b51]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260429 14:04:43.442009 11447 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3efb00c93ab44cf4864b468153d75028. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:43.442009 11163 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3efb00c93ab44cf4864b468153d75028. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:43.443455 11544 tablet_bootstrap.cc:492] T 3efb00c93ab44cf4864b468153d75028 P f49e2dd467f042f08da4345b01c63e9b: Bootstrap starting.
I20260429 14:04:43.443528 11305 tablet_service.cc:1511] Processing CreateTablet for tablet 3efb00c93ab44cf4864b468153d75028 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d572e29aab024f6cb916fbe5fab49b51]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260429 14:04:43.443678 11305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3efb00c93ab44cf4864b468153d75028. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:43.443879 11544 tablet_bootstrap.cc:654] T 3efb00c93ab44cf4864b468153d75028 P f49e2dd467f042f08da4345b01c63e9b: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:43.444414 11544 tablet_bootstrap.cc:492] T 3efb00c93ab44cf4864b468153d75028 P f49e2dd467f042f08da4345b01c63e9b: No bootstrap required, opened a new log
I20260429 14:04:43.444444 11544 ts_tablet_manager.cc:1403] T 3efb00c93ab44cf4864b468153d75028 P f49e2dd467f042f08da4345b01c63e9b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:43.444528 11544 raft_consensus.cc:359] T 3efb00c93ab44cf4864b468153d75028 P f49e2dd467f042f08da4345b01c63e9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:43.444573 11544 raft_consensus.cc:385] T 3efb00c93ab44cf4864b468153d75028 P f49e2dd467f042f08da4345b01c63e9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:43.444586 11544 raft_consensus.cc:740] T 3efb00c93ab44cf4864b468153d75028 P f49e2dd467f042f08da4345b01c63e9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f49e2dd467f042f08da4345b01c63e9b, State: Initialized, Role: FOLLOWER
I20260429 14:04:43.444653 11544 consensus_queue.cc:260] T 3efb00c93ab44cf4864b468153d75028 P f49e2dd467f042f08da4345b01c63e9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:43.444741 11544 ts_tablet_manager.cc:1434] T 3efb00c93ab44cf4864b468153d75028 P f49e2dd467f042f08da4345b01c63e9b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:43.444823 11542 tablet_bootstrap.cc:492] T 3efb00c93ab44cf4864b468153d75028 P 934c8c1fa4f64ba7941168f5ab504ce7: Bootstrap starting.
I20260429 14:04:43.444841 11541 tablet_bootstrap.cc:492] T 3efb00c93ab44cf4864b468153d75028 P 49573166d38746089e79505c86c6e93a: Bootstrap starting.
I20260429 14:04:43.444806 11551 tablet_replica.cc:442] T 3efb00c93ab44cf4864b468153d75028 P f49e2dd467f042f08da4345b01c63e9b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } } }
I20260429 14:04:43.444888 11551 tablet_replica.cc:445] T 3efb00c93ab44cf4864b468153d75028 P f49e2dd467f042f08da4345b01c63e9b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 14:04:43.445348 11542 tablet_bootstrap.cc:654] T 3efb00c93ab44cf4864b468153d75028 P 934c8c1fa4f64ba7941168f5ab504ce7: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:43.445392 11541 tablet_bootstrap.cc:654] T 3efb00c93ab44cf4864b468153d75028 P 49573166d38746089e79505c86c6e93a: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:43.446022 11542 tablet_bootstrap.cc:492] T 3efb00c93ab44cf4864b468153d75028 P 934c8c1fa4f64ba7941168f5ab504ce7: No bootstrap required, opened a new log
I20260429 14:04:43.446089 11542 ts_tablet_manager.cc:1403] T 3efb00c93ab44cf4864b468153d75028 P 934c8c1fa4f64ba7941168f5ab504ce7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:43.446252 11541 tablet_bootstrap.cc:492] T 3efb00c93ab44cf4864b468153d75028 P 49573166d38746089e79505c86c6e93a: No bootstrap required, opened a new log
I20260429 14:04:43.446247 11542 raft_consensus.cc:359] T 3efb00c93ab44cf4864b468153d75028 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:43.446321 11541 ts_tablet_manager.cc:1403] T 3efb00c93ab44cf4864b468153d75028 P 49573166d38746089e79505c86c6e93a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:43.446354 11542 raft_consensus.cc:385] T 3efb00c93ab44cf4864b468153d75028 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:43.446384 11542 raft_consensus.cc:740] T 3efb00c93ab44cf4864b468153d75028 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 934c8c1fa4f64ba7941168f5ab504ce7, State: Initialized, Role: FOLLOWER
I20260429 14:04:43.446440 11542 consensus_queue.cc:260] T 3efb00c93ab44cf4864b468153d75028 P 934c8c1fa4f64ba7941168f5ab504ce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:43.446475 11541 raft_consensus.cc:359] T 3efb00c93ab44cf4864b468153d75028 P 49573166d38746089e79505c86c6e93a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:43.446537 11542 ts_tablet_manager.cc:1434] T 3efb00c93ab44cf4864b468153d75028 P 934c8c1fa4f64ba7941168f5ab504ce7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:43.446568 11541 raft_consensus.cc:385] T 3efb00c93ab44cf4864b468153d75028 P 49573166d38746089e79505c86c6e93a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:43.446593 11541 raft_consensus.cc:740] T 3efb00c93ab44cf4864b468153d75028 P 49573166d38746089e79505c86c6e93a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49573166d38746089e79505c86c6e93a, State: Initialized, Role: FOLLOWER
I20260429 14:04:43.446580 11548 tablet_replica.cc:442] T 3efb00c93ab44cf4864b468153d75028 P 934c8c1fa4f64ba7941168f5ab504ce7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } } }
I20260429 14:04:43.446674 11548 tablet_replica.cc:445] T 3efb00c93ab44cf4864b468153d75028 P 934c8c1fa4f64ba7941168f5ab504ce7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 14:04:43.446673 11541 consensus_queue.cc:260] T 3efb00c93ab44cf4864b468153d75028 P 49573166d38746089e79505c86c6e93a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:43.446781 11541 ts_tablet_manager.cc:1434] T 3efb00c93ab44cf4864b468153d75028 P 49573166d38746089e79505c86c6e93a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:43.446799 11547 tablet_replica.cc:442] T 3efb00c93ab44cf4864b468153d75028 P 49573166d38746089e79505c86c6e93a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } } }
I20260429 14:04:43.446853 11547 tablet_replica.cc:445] T 3efb00c93ab44cf4864b468153d75028 P 49573166d38746089e79505c86c6e93a: This TxnStatusTablet replica's current role is: FOLLOWER
W20260429 14:04:43.542384 11514 tablet.cc:2404] T 2473765fd7df4b3d93d22e582f10b126 P f49e2dd467f042f08da4345b01c63e9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 14:04:43.606433 11548 raft_consensus.cc:493] T 3efb00c93ab44cf4864b468153d75028 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:04:43.606572 11548 raft_consensus.cc:515] T 3efb00c93ab44cf4864b468153d75028 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:43.606935 11548 leader_election.cc:290] T 3efb00c93ab44cf4864b468153d75028 P 934c8c1fa4f64ba7941168f5ab504ce7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49573166d38746089e79505c86c6e93a (127.2.197.193:42967), f49e2dd467f042f08da4345b01c63e9b (127.2.197.195:37495)
I20260429 14:04:43.609261 11583 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:43.607163 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:42967 (local address 127.2.197.194:47505)
0429 14:04:43.607305 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:43.607316 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:43.607406 (+ 90us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:43.607569 (+ 163us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:43.607570 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:43.607572 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:04:43.607788 (+ 216us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:43.607790 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.608420 (+ 630us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:43.608421 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:43.609011 (+ 590us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:43.609013 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.609107 (+ 94us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:43.609112 (+ 5us) client_negotiation.cc:770] Sending connection context
0429 14:04:43.609144 (+ 32us) client_negotiation.cc:241] Negotiation successful
0429 14:04:43.609179 (+ 35us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":82,"threads_started":1}
I20260429 14:04:43.609637 11536 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:43.607236 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:47505 (local address 127.2.197.193:42967)
0429 14:04:43.607288 (+ 52us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:43.607290 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:43.607397 (+ 107us) server_negotiation.cc:408] Connection header received
0429 14:04:43.607469 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:43.607470 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:43.607484 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:43.607552 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:04:43.607869 (+ 317us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.608324 (+ 455us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:43.609106 (+ 782us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.609540 (+ 434us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:43.609548 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:43.609567 (+ 19us) server_negotiation.cc:300] Negotiation successful
0429 14:04:43.609591 (+ 24us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260429 14:04:43.609865 11183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3efb00c93ab44cf4864b468153d75028" candidate_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49573166d38746089e79505c86c6e93a" is_pre_election: true
I20260429 14:04:43.609961 11183 raft_consensus.cc:2468] T 3efb00c93ab44cf4864b468153d75028 P 49573166d38746089e79505c86c6e93a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 934c8c1fa4f64ba7941168f5ab504ce7 in term 0.
I20260429 14:04:43.610030 11584 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:43.607209 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:37495 (local address 127.2.197.194:48711)
0429 14:04:43.607351 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:43.607360 (+ 9us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:43.607445 (+ 85us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:43.607631 (+ 186us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:43.607632 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:43.607635 (+ 3us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:04:43.607829 (+ 194us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:43.607830 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.608647 (+ 817us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:43.608648 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:43.609227 (+ 579us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:43.609229 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.609864 (+ 635us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:43.609868 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 14:04:43.609916 (+ 48us) client_negotiation.cc:241] Negotiation successful
0429 14:04:43.609958 (+ 42us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":77,"threads_started":1}
I20260429 14:04:43.610040 11539 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:43.607279 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:48711 (local address 127.2.197.195:37495)
0429 14:04:43.607322 (+ 43us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:43.607324 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:43.607431 (+ 107us) server_negotiation.cc:408] Connection header received
0429 14:04:43.607513 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:43.607515 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:43.607525 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:43.607580 (+ 55us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:04:43.607917 (+ 337us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.608523 (+ 606us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:43.609353 (+ 830us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.609833 (+ 480us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:43.609885 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:43.609922 (+ 37us) server_negotiation.cc:300] Negotiation successful
0429 14:04:43.609963 (+ 41us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260429 14:04:43.610127 11258 leader_election.cc:304] T 3efb00c93ab44cf4864b468153d75028 P 934c8c1fa4f64ba7941168f5ab504ce7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49573166d38746089e79505c86c6e93a, 934c8c1fa4f64ba7941168f5ab504ce7; no voters:
I20260429 14:04:43.610301 11548 raft_consensus.cc:2804] T 3efb00c93ab44cf4864b468153d75028 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 14:04:43.610350 11548 raft_consensus.cc:493] T 3efb00c93ab44cf4864b468153d75028 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 14:04:43.610375 11548 raft_consensus.cc:3060] T 3efb00c93ab44cf4864b468153d75028 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:43.610348 11467 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3efb00c93ab44cf4864b468153d75028" candidate_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f49e2dd467f042f08da4345b01c63e9b" is_pre_election: true
I20260429 14:04:43.610420 11467 raft_consensus.cc:2468] T 3efb00c93ab44cf4864b468153d75028 P f49e2dd467f042f08da4345b01c63e9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 934c8c1fa4f64ba7941168f5ab504ce7 in term 0.
I20260429 14:04:43.611035 11548 raft_consensus.cc:515] T 3efb00c93ab44cf4864b468153d75028 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:43.611181 11548 leader_election.cc:290] T 3efb00c93ab44cf4864b468153d75028 P 934c8c1fa4f64ba7941168f5ab504ce7 [CANDIDATE]: Term 1 election: Requested vote from peers 49573166d38746089e79505c86c6e93a (127.2.197.193:42967), f49e2dd467f042f08da4345b01c63e9b (127.2.197.195:37495)
I20260429 14:04:43.611352 11467 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3efb00c93ab44cf4864b468153d75028" candidate_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f49e2dd467f042f08da4345b01c63e9b"
I20260429 14:04:43.611366 11183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3efb00c93ab44cf4864b468153d75028" candidate_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49573166d38746089e79505c86c6e93a"
I20260429 14:04:43.611438 11183 raft_consensus.cc:3060] T 3efb00c93ab44cf4864b468153d75028 P 49573166d38746089e79505c86c6e93a [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:43.611442 11467 raft_consensus.cc:3060] T 3efb00c93ab44cf4864b468153d75028 P f49e2dd467f042f08da4345b01c63e9b [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:43.612023 11183 raft_consensus.cc:2468] T 3efb00c93ab44cf4864b468153d75028 P 49573166d38746089e79505c86c6e93a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 934c8c1fa4f64ba7941168f5ab504ce7 in term 1.
I20260429 14:04:43.612104 11467 raft_consensus.cc:2468] T 3efb00c93ab44cf4864b468153d75028 P f49e2dd467f042f08da4345b01c63e9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 934c8c1fa4f64ba7941168f5ab504ce7 in term 1.
I20260429 14:04:43.612211 11258 leader_election.cc:304] T 3efb00c93ab44cf4864b468153d75028 P 934c8c1fa4f64ba7941168f5ab504ce7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49573166d38746089e79505c86c6e93a, 934c8c1fa4f64ba7941168f5ab504ce7; no voters:
I20260429 14:04:43.612330 11548 raft_consensus.cc:2804] T 3efb00c93ab44cf4864b468153d75028 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:04:43.612466 11548 raft_consensus.cc:697] T 3efb00c93ab44cf4864b468153d75028 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 1 LEADER]: Becoming Leader. State: Replica: 934c8c1fa4f64ba7941168f5ab504ce7, State: Running, Role: LEADER
I20260429 14:04:43.612610 11548 consensus_queue.cc:237] T 3efb00c93ab44cf4864b468153d75028 P 934c8c1fa4f64ba7941168f5ab504ce7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:43.613044 11585 tablet_replica.cc:442] T 3efb00c93ab44cf4864b468153d75028 P 934c8c1fa4f64ba7941168f5ab504ce7: TxnStatusTablet state changed. Reason: New leader 934c8c1fa4f64ba7941168f5ab504ce7. Latest consensus state: current_term: 1 leader_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } } }
I20260429 14:04:43.613130 11585 tablet_replica.cc:445] T 3efb00c93ab44cf4864b468153d75028 P 934c8c1fa4f64ba7941168f5ab504ce7: This TxnStatusTablet replica's current role is: LEADER
I20260429 14:04:43.613297 11587 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260429 14:04:43.613392 11587 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260429 14:04:43.613433 11024 catalog_manager.cc:5671] T 3efb00c93ab44cf4864b468153d75028 P 934c8c1fa4f64ba7941168f5ab504ce7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 934c8c1fa4f64ba7941168f5ab504ce7 (127.2.197.194). New cstate: current_term: 1 leader_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } health_report { overall_health: UNKNOWN } } }
I20260429 14:04:43.634578 11579 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:43.632318 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:37361 (local address 127.0.0.1:38702)
0429 14:04:43.632395 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:43.632405 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:43.632474 (+ 69us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:43.632687 (+ 213us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:43.632688 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:43.632689 (+ 1us) client_negotiation.cc:190] Negotiated authn=TOKEN
0429 14:04:43.632852 (+ 163us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:43.632853 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.633459 (+ 606us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:43.633460 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:43.633890 (+ 430us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:43.633891 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.633979 (+ 88us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:43.633994 (+ 15us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0429 14:04:43.634384 (+ 390us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0429 14:04:43.634392 (+ 8us) client_negotiation.cc:770] Sending connection context
0429 14:04:43.634461 (+ 69us) client_negotiation.cc:241] Negotiation successful
0429 14:04:43.634514 (+ 53us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260429 14:04:43.634574 11540 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:43.632396 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:38702 (local address 127.2.197.194:37361)
0429 14:04:43.632461 (+ 65us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:43.632463 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:43.632472 (+ 9us) server_negotiation.cc:408] Connection header received
0429 14:04:43.632593 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:43.632594 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:43.632608 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:43.632678 (+ 70us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:04:43.632934 (+ 256us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.633378 (+ 444us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:43.633977 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.634136 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:43.634161 (+ 25us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:04:43.634238 (+ 77us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:04:43.634376 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:43.634465 (+ 89us) server_negotiation.cc:300] Negotiation successful
0429 14:04:43.634490 (+ 25us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260429 14:04:43.635740 11183 raft_consensus.cc:1275] T 3efb00c93ab44cf4864b468153d75028 P 49573166d38746089e79505c86c6e93a [term 1 FOLLOWER]: Refusing update from remote peer 934c8c1fa4f64ba7941168f5ab504ce7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:04:43.635880 11467 raft_consensus.cc:1275] T 3efb00c93ab44cf4864b468153d75028 P f49e2dd467f042f08da4345b01c63e9b [term 1 FOLLOWER]: Refusing update from remote peer 934c8c1fa4f64ba7941168f5ab504ce7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:04:43.636041 11585 consensus_queue.cc:1048] T 3efb00c93ab44cf4864b468153d75028 P 934c8c1fa4f64ba7941168f5ab504ce7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:04:43.636149 11548 consensus_queue.cc:1048] T 3efb00c93ab44cf4864b468153d75028 P 934c8c1fa4f64ba7941168f5ab504ce7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:04:43.636770 11547 tablet_replica.cc:442] T 3efb00c93ab44cf4864b468153d75028 P 49573166d38746089e79505c86c6e93a: TxnStatusTablet state changed. Reason: New leader 934c8c1fa4f64ba7941168f5ab504ce7. Latest consensus state: current_term: 1 leader_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } } }
I20260429 14:04:43.636775 11551 tablet_replica.cc:442] T 3efb00c93ab44cf4864b468153d75028 P f49e2dd467f042f08da4345b01c63e9b: TxnStatusTablet state changed. Reason: New leader 934c8c1fa4f64ba7941168f5ab504ce7. Latest consensus state: current_term: 1 leader_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } } }
I20260429 14:04:43.636868 11547 tablet_replica.cc:445] T 3efb00c93ab44cf4864b468153d75028 P 49573166d38746089e79505c86c6e93a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 14:04:43.636868 11551 tablet_replica.cc:445] T 3efb00c93ab44cf4864b468153d75028 P f49e2dd467f042f08da4345b01c63e9b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 14:04:43.637657 11548 tablet_replica.cc:442] T 3efb00c93ab44cf4864b468153d75028 P 934c8c1fa4f64ba7941168f5ab504ce7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } } }
I20260429 14:04:43.637742 11548 tablet_replica.cc:445] T 3efb00c93ab44cf4864b468153d75028 P 934c8c1fa4f64ba7941168f5ab504ce7: This TxnStatusTablet replica's current role is: LEADER
I20260429 14:04:43.637741 11554 mvcc.cc:204] Tried to move back new op lower bound from 7280523196969594880 to 7280523196877832192. Current Snapshot: MvccSnapshot[applied={T|T < 7280523196969594880}]
I20260429 14:04:43.638005 11551 tablet_replica.cc:442] T 3efb00c93ab44cf4864b468153d75028 P f49e2dd467f042f08da4345b01c63e9b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } } }
I20260429 14:04:43.638077 11551 tablet_replica.cc:445] T 3efb00c93ab44cf4864b468153d75028 P f49e2dd467f042f08da4345b01c63e9b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 14:04:43.638962 11547 tablet_replica.cc:442] T 3efb00c93ab44cf4864b468153d75028 P 49573166d38746089e79505c86c6e93a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } } }
I20260429 14:04:43.639055 11547 tablet_replica.cc:445] T 3efb00c93ab44cf4864b468153d75028 P 49573166d38746089e79505c86c6e93a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 14:04:43.638813 11585 tablet_replica.cc:442] T 3efb00c93ab44cf4864b468153d75028 P 934c8c1fa4f64ba7941168f5ab504ce7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } } }
I20260429 14:04:43.639256 11585 tablet_replica.cc:445] T 3efb00c93ab44cf4864b468153d75028 P 934c8c1fa4f64ba7941168f5ab504ce7: This TxnStatusTablet replica's current role is: LEADER
I20260429 14:04:43.641548 11536 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:43.639243 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:39608 (local address 127.2.197.193:42967)
0429 14:04:43.639285 (+ 42us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:43.639287 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:43.639295 (+ 8us) server_negotiation.cc:408] Connection header received
0429 14:04:43.639337 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:43.639338 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:43.639352 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:43.639405 (+ 53us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:04:43.639747 (+ 342us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.640204 (+ 457us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:43.641053 (+ 849us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.641186 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:43.641218 (+ 32us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:04:43.641303 (+ 85us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:04:43.641361 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:43.641442 (+ 81us) server_negotiation.cc:300] Negotiation successful
0429 14:04:43.641464 (+ 22us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260429 14:04:43.646701 11595 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:43.644145 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:37361 (local address 127.2.197.193:58667)
0429 14:04:43.644327 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:43.644336 (+ 9us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:43.644396 (+ 60us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:43.644555 (+ 159us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:43.644556 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:43.644557 (+ 1us) client_negotiation.cc:190] Negotiated authn=TOKEN
0429 14:04:43.644736 (+ 179us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:43.644738 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.645641 (+ 903us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:43.645642 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:43.646098 (+ 456us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:43.646099 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.646165 (+ 66us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:43.646178 (+ 13us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0429 14:04:43.646518 (+ 340us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0429 14:04:43.646521 (+ 3us) client_negotiation.cc:770] Sending connection context
0429 14:04:43.646580 (+ 59us) client_negotiation.cc:241] Negotiation successful
0429 14:04:43.646618 (+ 38us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":89,"threads_started":1}
I20260429 14:04:43.646696 11540 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:43.644196 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:58667 (local address 127.2.197.194:37361)
0429 14:04:43.644269 (+ 73us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:43.644270 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:43.644407 (+ 137us) server_negotiation.cc:408] Connection header received
0429 14:04:43.644437 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:43.644438 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:43.644449 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:43.644510 (+ 61us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:04:43.644813 (+ 303us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.645541 (+ 728us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:43.646180 (+ 639us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.646389 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:43.646411 (+ 22us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:04:43.646438 (+ 27us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:04:43.646499 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:43.646602 (+ 103us) server_negotiation.cc:300] Negotiation successful
0429 14:04:43.646624 (+ 22us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260429 14:04:43.649166 11467 raft_consensus.cc:1275] T 2473765fd7df4b3d93d22e582f10b126 P f49e2dd467f042f08da4345b01c63e9b [term 1 FOLLOWER]: Refusing update from remote peer 49573166d38746089e79505c86c6e93a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:04:43.649199 11325 raft_consensus.cc:1275] T 2473765fd7df4b3d93d22e582f10b126 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 1 FOLLOWER]: Refusing update from remote peer 49573166d38746089e79505c86c6e93a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:04:43.649422 11572 consensus_queue.cc:1048] T 2473765fd7df4b3d93d22e582f10b126 P 49573166d38746089e79505c86c6e93a [LEADER]: Connected to new peer: Peer: permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:04:43.649575 11572 consensus_queue.cc:1048] T 2473765fd7df4b3d93d22e582f10b126 P 49573166d38746089e79505c86c6e93a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:04:43.656636 11556 mvcc.cc:204] Tried to move back new op lower bound from 7280523197025423360 to 7280523196010094592. Current Snapshot: MvccSnapshot[applied={T|T < 7280523197025423360}]
I20260429 14:04:43.658777 11600 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:43.656206 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:42967 (local address 127.2.197.194:35489)
0429 14:04:43.656366 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:43.656377 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:43.656460 (+ 83us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:43.656656 (+ 196us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:43.656657 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:43.656659 (+ 2us) client_negotiation.cc:190] Negotiated authn=TOKEN
0429 14:04:43.657007 (+ 348us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:43.657010 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.657701 (+ 691us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:43.657703 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:43.658133 (+ 430us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:43.658134 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.658217 (+ 83us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:43.658231 (+ 14us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0429 14:04:43.658547 (+ 316us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0429 14:04:43.658549 (+ 2us) client_negotiation.cc:770] Sending connection context
0429 14:04:43.658625 (+ 76us) client_negotiation.cc:241] Negotiation successful
0429 14:04:43.658680 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":87,"threads_started":1}
I20260429 14:04:43.658772 11536 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:43.656222 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:35489 (local address 127.2.197.193:42967)
0429 14:04:43.656357 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:43.656359 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:43.656436 (+ 77us) server_negotiation.cc:408] Connection header received
0429 14:04:43.656538 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:43.656539 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:43.656549 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:43.656581 (+ 32us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:04:43.657077 (+ 496us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.657621 (+ 544us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:43.658228 (+ 607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.658414 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:43.658433 (+ 19us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:04:43.658452 (+ 19us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:04:43.658532 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:43.658689 (+ 157us) server_negotiation.cc:300] Negotiation successful
0429 14:04:43.658711 (+ 22us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260429 14:04:43.671445 11023 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:04:43.671581 11023 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:04:43.672305 11023 catalog_manager.cc:5958] T 00000000000000000000000000000000 P bb7ccf4eb15e4d9e84fe3634fa4b154b: Sending DeleteTablet for 3 replicas of tablet 2473765fd7df4b3d93d22e582f10b126
I20260429 14:04:43.672622 11163 tablet_service.cc:1558] Processing DeleteTablet for tablet 2473765fd7df4b3d93d22e582f10b126 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:04:43 UTC) from {username='oryx', principal='oryx/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:39604
I20260429 14:04:43.672645 11447 tablet_service.cc:1558] Processing DeleteTablet for tablet 2473765fd7df4b3d93d22e582f10b126 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:04:43 UTC) from {username='oryx', principal='oryx/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:37490
I20260429 14:04:43.672716 11564 tablet_replica.cc:333] T 2473765fd7df4b3d93d22e582f10b126 P 49573166d38746089e79505c86c6e93a: stopping tablet replica
I20260429 14:04:43.672750 11565 tablet_replica.cc:333] T 2473765fd7df4b3d93d22e582f10b126 P f49e2dd467f042f08da4345b01c63e9b: stopping tablet replica
I20260429 14:04:43.672770 11564 raft_consensus.cc:2243] T 2473765fd7df4b3d93d22e582f10b126 P 49573166d38746089e79505c86c6e93a [term 1 LEADER]: Raft consensus shutting down.
I20260429 14:04:43.672801 11565 raft_consensus.cc:2243] T 2473765fd7df4b3d93d22e582f10b126 P f49e2dd467f042f08da4345b01c63e9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:04:43.672853 11564 raft_consensus.cc:2272] T 2473765fd7df4b3d93d22e582f10b126 P 49573166d38746089e79505c86c6e93a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:04:43.672875 11565 raft_consensus.cc:2272] T 2473765fd7df4b3d93d22e582f10b126 P f49e2dd467f042f08da4345b01c63e9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:04:43.673215 11564 ts_tablet_manager.cc:1916] T 2473765fd7df4b3d93d22e582f10b126 P 49573166d38746089e79505c86c6e93a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:04:43.673353 11565 ts_tablet_manager.cc:1916] T 2473765fd7df4b3d93d22e582f10b126 P f49e2dd467f042f08da4345b01c63e9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:04:43.674961 11565 ts_tablet_manager.cc:1929] T 2473765fd7df4b3d93d22e582f10b126 P f49e2dd467f042f08da4345b01c63e9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 14:04:43.674983 11305 tablet_service.cc:1558] Processing DeleteTablet for tablet 2473765fd7df4b3d93d22e582f10b126 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:04:43 UTC) from {username='oryx', principal='oryx/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:38694
I20260429 14:04:43.675016 11565 log.cc:1199] T 2473765fd7df4b3d93d22e582f10b126 P f49e2dd467f042f08da4345b01c63e9b: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-2/wal/wals/2473765fd7df4b3d93d22e582f10b126
I20260429 14:04:43.675087 11566 tablet_replica.cc:333] T 2473765fd7df4b3d93d22e582f10b126 P 934c8c1fa4f64ba7941168f5ab504ce7: stopping tablet replica
I20260429 14:04:43.675146 11566 raft_consensus.cc:2243] T 2473765fd7df4b3d93d22e582f10b126 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:04:43.675192 11566 raft_consensus.cc:2272] T 2473765fd7df4b3d93d22e582f10b126 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:04:43.675232 11565 ts_tablet_manager.cc:1950] T 2473765fd7df4b3d93d22e582f10b126 P f49e2dd467f042f08da4345b01c63e9b: Deleting consensus metadata
I20260429 14:04:43.676023 11566 ts_tablet_manager.cc:1916] T 2473765fd7df4b3d93d22e582f10b126 P 934c8c1fa4f64ba7941168f5ab504ce7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:04:43.677301 11008 catalog_manager.cc:5002] TS f49e2dd467f042f08da4345b01c63e9b (127.2.197.195:37495): tablet 2473765fd7df4b3d93d22e582f10b126 (table test-table [id=823a76f281ac4203aa3ac9c65bbd069b]) successfully deleted
I20260429 14:04:43.677441 11566 ts_tablet_manager.cc:1929] T 2473765fd7df4b3d93d22e582f10b126 P 934c8c1fa4f64ba7941168f5ab504ce7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 14:04:43.677490 11566 log.cc:1199] T 2473765fd7df4b3d93d22e582f10b126 P 934c8c1fa4f64ba7941168f5ab504ce7: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-1/wal/wals/2473765fd7df4b3d93d22e582f10b126
I20260429 14:04:43.677834 11566 ts_tablet_manager.cc:1950] T 2473765fd7df4b3d93d22e582f10b126 P 934c8c1fa4f64ba7941168f5ab504ce7: Deleting consensus metadata
I20260429 14:04:43.678311 11009 catalog_manager.cc:5002] TS 934c8c1fa4f64ba7941168f5ab504ce7 (127.2.197.194:37361): tablet 2473765fd7df4b3d93d22e582f10b126 (table test-table [id=823a76f281ac4203aa3ac9c65bbd069b]) successfully deleted
I20260429 14:04:43.678397 11564 ts_tablet_manager.cc:1929] T 2473765fd7df4b3d93d22e582f10b126 P 49573166d38746089e79505c86c6e93a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 14:04:43.678454 11564 log.cc:1199] T 2473765fd7df4b3d93d22e582f10b126 P 49573166d38746089e79505c86c6e93a: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-0/wal/wals/2473765fd7df4b3d93d22e582f10b126
I20260429 14:04:43.678663 11564 ts_tablet_manager.cc:1950] T 2473765fd7df4b3d93d22e582f10b126 P 49573166d38746089e79505c86c6e93a: Deleting consensus metadata
I20260429 14:04:43.679816 11010 catalog_manager.cc:5002] TS 49573166d38746089e79505c86c6e93a (127.2.197.193:42967): tablet 2473765fd7df4b3d93d22e582f10b126 (table test-table [id=823a76f281ac4203aa3ac9c65bbd069b]) successfully deleted
Apr 29 14:04:43 dist-test-slave-8n77 krb5kdc[10976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Apr 29 14:04:43 dist-test-slave-8n77 krb5kdc[10976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:04:43 dist-test-slave-8n77 krb5kdc[10976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
W20260429 14:04:43.707394 11073 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:43.700643 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57838 (local address 127.2.197.254:42715)
0429 14:04:43.700692 (+ 49us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:43.700695 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:43.700835 (+ 140us) server_negotiation.cc:408] Connection header received
0429 14:04:43.700890 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:43.700892 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:43.700904 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:43.700985 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:43.701949 (+ 964us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.702874 (+ 925us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:43.703743 (+ 869us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.704286 (+ 543us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:43.707311 (+ 3025us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57838: BlockingRecv error: recv got EOF from 127.0.0.1:57838 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Apr 29 14:04:43 dist-test-slave-8n77 krb5kdc[10976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM
I20260429 14:04:43.717561 11073 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:43.708684 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57840 (local address 127.2.197.254:42715)
0429 14:04:43.708728 (+ 44us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:43.708730 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:43.708981 (+ 251us) server_negotiation.cc:408] Connection header received
0429 14:04:43.709053 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:43.709054 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:43.709066 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:43.709121 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:43.709924 (+ 803us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.710729 (+ 805us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:43.711594 (+ 865us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:43.711780 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:43.714563 (+ 2783us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:43.714568 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:43.714569 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:43.714579 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:43.716353 (+ 1774us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:43.716758 (+ 405us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:43.716760 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:43.716760 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:43.716815 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:43.716989 (+ 174us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:43.716990 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:43.716990 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:43.717118 (+ 128us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:43.717227 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:43.717353 (+ 126us) server_negotiation.cc:300] Negotiation successful
0429 14:04:43.717453 (+ 100us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260429 14:04:43.719609 11023 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 14:04:43.719798 11023 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 14:04:43.721568 11447 tablet_service.cc:1511] Processing CreateTablet for tablet 9cb826017c9347e48bf70204b83b9370 (DEFAULT_TABLE table=test-table [id=ae913a4988f34bf1b1760543f5952f80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:04:43.721714 11447 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cb826017c9347e48bf70204b83b9370. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:43.723206 11163 tablet_service.cc:1511] Processing CreateTablet for tablet 9cb826017c9347e48bf70204b83b9370 (DEFAULT_TABLE table=test-table [id=ae913a4988f34bf1b1760543f5952f80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:04:43.723335 11163 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cb826017c9347e48bf70204b83b9370. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:43.724342 11544 tablet_bootstrap.cc:492] T 9cb826017c9347e48bf70204b83b9370 P f49e2dd467f042f08da4345b01c63e9b: Bootstrap starting.
I20260429 14:04:43.724748 11544 tablet_bootstrap.cc:654] T 9cb826017c9347e48bf70204b83b9370 P f49e2dd467f042f08da4345b01c63e9b: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:43.725076 11305 tablet_service.cc:1511] Processing CreateTablet for tablet 9cb826017c9347e48bf70204b83b9370 (DEFAULT_TABLE table=test-table [id=ae913a4988f34bf1b1760543f5952f80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:04:43.725186 11305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cb826017c9347e48bf70204b83b9370. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:43.725191 11541 tablet_bootstrap.cc:492] T 9cb826017c9347e48bf70204b83b9370 P 49573166d38746089e79505c86c6e93a: Bootstrap starting.
I20260429 14:04:43.725605 11541 tablet_bootstrap.cc:654] T 9cb826017c9347e48bf70204b83b9370 P 49573166d38746089e79505c86c6e93a: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:43.726668 11544 tablet_bootstrap.cc:492] T 9cb826017c9347e48bf70204b83b9370 P f49e2dd467f042f08da4345b01c63e9b: No bootstrap required, opened a new log
I20260429 14:04:43.726747 11544 ts_tablet_manager.cc:1403] T 9cb826017c9347e48bf70204b83b9370 P f49e2dd467f042f08da4345b01c63e9b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260429 14:04:43.726874 11544 raft_consensus.cc:359] T 9cb826017c9347e48bf70204b83b9370 P f49e2dd467f042f08da4345b01c63e9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:43.726946 11544 raft_consensus.cc:385] T 9cb826017c9347e48bf70204b83b9370 P f49e2dd467f042f08da4345b01c63e9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:43.726966 11544 raft_consensus.cc:740] T 9cb826017c9347e48bf70204b83b9370 P f49e2dd467f042f08da4345b01c63e9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f49e2dd467f042f08da4345b01c63e9b, State: Initialized, Role: FOLLOWER
I20260429 14:04:43.727010 11544 consensus_queue.cc:260] T 9cb826017c9347e48bf70204b83b9370 P f49e2dd467f042f08da4345b01c63e9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:43.727092 11544 ts_tablet_manager.cc:1434] T 9cb826017c9347e48bf70204b83b9370 P f49e2dd467f042f08da4345b01c63e9b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260429 14:04:43.727658 11542 tablet_bootstrap.cc:492] T 9cb826017c9347e48bf70204b83b9370 P 934c8c1fa4f64ba7941168f5ab504ce7: Bootstrap starting.
I20260429 14:04:43.727988 11542 tablet_bootstrap.cc:654] T 9cb826017c9347e48bf70204b83b9370 P 934c8c1fa4f64ba7941168f5ab504ce7: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:43.728281 11541 tablet_bootstrap.cc:492] T 9cb826017c9347e48bf70204b83b9370 P 49573166d38746089e79505c86c6e93a: No bootstrap required, opened a new log
I20260429 14:04:43.728331 11541 ts_tablet_manager.cc:1403] T 9cb826017c9347e48bf70204b83b9370 P 49573166d38746089e79505c86c6e93a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260429 14:04:43.728639 11541 raft_consensus.cc:359] T 9cb826017c9347e48bf70204b83b9370 P 49573166d38746089e79505c86c6e93a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:43.728729 11541 raft_consensus.cc:385] T 9cb826017c9347e48bf70204b83b9370 P 49573166d38746089e79505c86c6e93a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:43.728765 11541 raft_consensus.cc:740] T 9cb826017c9347e48bf70204b83b9370 P 49573166d38746089e79505c86c6e93a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49573166d38746089e79505c86c6e93a, State: Initialized, Role: FOLLOWER
I20260429 14:04:43.728818 11541 consensus_queue.cc:260] T 9cb826017c9347e48bf70204b83b9370 P 49573166d38746089e79505c86c6e93a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:43.728848 11542 tablet_bootstrap.cc:492] T 9cb826017c9347e48bf70204b83b9370 P 934c8c1fa4f64ba7941168f5ab504ce7: No bootstrap required, opened a new log
I20260429 14:04:43.728917 11541 ts_tablet_manager.cc:1434] T 9cb826017c9347e48bf70204b83b9370 P 49573166d38746089e79505c86c6e93a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:43.728904 11542 ts_tablet_manager.cc:1403] T 9cb826017c9347e48bf70204b83b9370 P 934c8c1fa4f64ba7941168f5ab504ce7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:43.729074 11542 raft_consensus.cc:359] T 9cb826017c9347e48bf70204b83b9370 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:43.729140 11542 raft_consensus.cc:385] T 9cb826017c9347e48bf70204b83b9370 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:43.729161 11542 raft_consensus.cc:740] T 9cb826017c9347e48bf70204b83b9370 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 934c8c1fa4f64ba7941168f5ab504ce7, State: Initialized, Role: FOLLOWER
I20260429 14:04:43.729204 11542 consensus_queue.cc:260] T 9cb826017c9347e48bf70204b83b9370 P 934c8c1fa4f64ba7941168f5ab504ce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:43.729291 11542 ts_tablet_manager.cc:1434] T 9cb826017c9347e48bf70204b83b9370 P 934c8c1fa4f64ba7941168f5ab504ce7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:43.996397 11592 raft_consensus.cc:493] T 9cb826017c9347e48bf70204b83b9370 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:04:43.996539 11592 raft_consensus.cc:515] T 9cb826017c9347e48bf70204b83b9370 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:43.996771 11592 leader_election.cc:290] T 9cb826017c9347e48bf70204b83b9370 P 934c8c1fa4f64ba7941168f5ab504ce7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49573166d38746089e79505c86c6e93a (127.2.197.193:42967), f49e2dd467f042f08da4345b01c63e9b (127.2.197.195:37495)
I20260429 14:04:43.997051 11183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cb826017c9347e48bf70204b83b9370" candidate_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49573166d38746089e79505c86c6e93a" is_pre_election: true
I20260429 14:04:43.997067 11467 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cb826017c9347e48bf70204b83b9370" candidate_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f49e2dd467f042f08da4345b01c63e9b" is_pre_election: true
I20260429 14:04:43.997169 11183 raft_consensus.cc:2468] T 9cb826017c9347e48bf70204b83b9370 P 49573166d38746089e79505c86c6e93a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 934c8c1fa4f64ba7941168f5ab504ce7 in term 0.
I20260429 14:04:43.997164 11467 raft_consensus.cc:2468] T 9cb826017c9347e48bf70204b83b9370 P f49e2dd467f042f08da4345b01c63e9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 934c8c1fa4f64ba7941168f5ab504ce7 in term 0.
I20260429 14:04:43.997383 11256 leader_election.cc:304] T 9cb826017c9347e48bf70204b83b9370 P 934c8c1fa4f64ba7941168f5ab504ce7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 934c8c1fa4f64ba7941168f5ab504ce7, f49e2dd467f042f08da4345b01c63e9b; no voters:
I20260429 14:04:43.997498 11592 raft_consensus.cc:2804] T 9cb826017c9347e48bf70204b83b9370 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 14:04:43.997545 11592 raft_consensus.cc:493] T 9cb826017c9347e48bf70204b83b9370 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 14:04:43.997565 11592 raft_consensus.cc:3060] T 9cb826017c9347e48bf70204b83b9370 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:43.998371 11592 raft_consensus.cc:515] T 9cb826017c9347e48bf70204b83b9370 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:43.998513 11592 leader_election.cc:290] T 9cb826017c9347e48bf70204b83b9370 P 934c8c1fa4f64ba7941168f5ab504ce7 [CANDIDATE]: Term 1 election: Requested vote from peers 49573166d38746089e79505c86c6e93a (127.2.197.193:42967), f49e2dd467f042f08da4345b01c63e9b (127.2.197.195:37495)
I20260429 14:04:43.998698 11467 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cb826017c9347e48bf70204b83b9370" candidate_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f49e2dd467f042f08da4345b01c63e9b"
I20260429 14:04:43.998764 11467 raft_consensus.cc:3060] T 9cb826017c9347e48bf70204b83b9370 P f49e2dd467f042f08da4345b01c63e9b [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:43.998719 11183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cb826017c9347e48bf70204b83b9370" candidate_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49573166d38746089e79505c86c6e93a"
I20260429 14:04:43.998843 11183 raft_consensus.cc:3060] T 9cb826017c9347e48bf70204b83b9370 P 49573166d38746089e79505c86c6e93a [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:43.999393 11467 raft_consensus.cc:2468] T 9cb826017c9347e48bf70204b83b9370 P f49e2dd467f042f08da4345b01c63e9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 934c8c1fa4f64ba7941168f5ab504ce7 in term 1.
I20260429 14:04:43.999521 11183 raft_consensus.cc:2468] T 9cb826017c9347e48bf70204b83b9370 P 49573166d38746089e79505c86c6e93a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 934c8c1fa4f64ba7941168f5ab504ce7 in term 1.
I20260429 14:04:43.999614 11256 leader_election.cc:304] T 9cb826017c9347e48bf70204b83b9370 P 934c8c1fa4f64ba7941168f5ab504ce7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 934c8c1fa4f64ba7941168f5ab504ce7, f49e2dd467f042f08da4345b01c63e9b; no voters:
I20260429 14:04:43.999712 11592 raft_consensus.cc:2804] T 9cb826017c9347e48bf70204b83b9370 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:04:43.999763 11592 raft_consensus.cc:697] T 9cb826017c9347e48bf70204b83b9370 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 1 LEADER]: Becoming Leader. State: Replica: 934c8c1fa4f64ba7941168f5ab504ce7, State: Running, Role: LEADER
I20260429 14:04:43.999810 11592 consensus_queue.cc:237] T 9cb826017c9347e48bf70204b83b9370 P 934c8c1fa4f64ba7941168f5ab504ce7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:44.000571 11023 catalog_manager.cc:5671] T 9cb826017c9347e48bf70204b83b9370 P 934c8c1fa4f64ba7941168f5ab504ce7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 934c8c1fa4f64ba7941168f5ab504ce7 (127.2.197.194). New cstate: current_term: 1 leader_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } health_report { overall_health: UNKNOWN } } }
I20260429 14:04:44.016101 11540 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:44.013710 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:38710 (local address 127.2.197.194:37361)
0429 14:04:44.013809 (+ 99us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:44.013811 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:44.013822 (+ 11us) server_negotiation.cc:408] Connection header received
0429 14:04:44.013846 (+ 24us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:44.013847 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:44.013856 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:44.013907 (+ 51us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:04:44.014241 (+ 334us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:44.014809 (+ 568us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:44.015517 (+ 708us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:44.015691 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:44.015710 (+ 19us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:04:44.015738 (+ 28us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:04:44.015797 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:44.016006 (+ 209us) server_negotiation.cc:300] Negotiation successful
0429 14:04:44.016044 (+ 38us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260429 14:04:44.017084 11467 raft_consensus.cc:1275] T 9cb826017c9347e48bf70204b83b9370 P f49e2dd467f042f08da4345b01c63e9b [term 1 FOLLOWER]: Refusing update from remote peer 934c8c1fa4f64ba7941168f5ab504ce7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:04:44.017102 11183 raft_consensus.cc:1275] T 9cb826017c9347e48bf70204b83b9370 P 49573166d38746089e79505c86c6e93a [term 1 FOLLOWER]: Refusing update from remote peer 934c8c1fa4f64ba7941168f5ab504ce7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:04:44.017369 11548 consensus_queue.cc:1048] T 9cb826017c9347e48bf70204b83b9370 P 934c8c1fa4f64ba7941168f5ab504ce7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:04:44.017532 11592 consensus_queue.cc:1048] T 9cb826017c9347e48bf70204b83b9370 P 934c8c1fa4f64ba7941168f5ab504ce7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:04:44.021895 11024 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:04:44.022056 11024 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:04:44.022981 11024 catalog_manager.cc:5958] T 00000000000000000000000000000000 P bb7ccf4eb15e4d9e84fe3634fa4b154b: Sending DeleteTablet for 3 replicas of tablet 9cb826017c9347e48bf70204b83b9370
I20260429 14:04:44.023296 11163 tablet_service.cc:1558] Processing DeleteTablet for tablet 9cb826017c9347e48bf70204b83b9370 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:04:44 UTC) from {username='oryx', principal='oryx/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:39604
I20260429 14:04:44.023301 11447 tablet_service.cc:1558] Processing DeleteTablet for tablet 9cb826017c9347e48bf70204b83b9370 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:04:44 UTC) from {username='oryx', principal='oryx/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:37490
I20260429 14:04:44.023399 11564 tablet_replica.cc:333] T 9cb826017c9347e48bf70204b83b9370 P 49573166d38746089e79505c86c6e93a: stopping tablet replica
I20260429 14:04:44.023386 11305 tablet_service.cc:1558] Processing DeleteTablet for tablet 9cb826017c9347e48bf70204b83b9370 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:04:44 UTC) from {username='oryx', principal='oryx/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:38694
I20260429 14:04:44.023411 11565 tablet_replica.cc:333] T 9cb826017c9347e48bf70204b83b9370 P f49e2dd467f042f08da4345b01c63e9b: stopping tablet replica
I20260429 14:04:44.023442 11564 raft_consensus.cc:2243] T 9cb826017c9347e48bf70204b83b9370 P 49573166d38746089e79505c86c6e93a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:04:44.023456 11565 raft_consensus.cc:2243] T 9cb826017c9347e48bf70204b83b9370 P f49e2dd467f042f08da4345b01c63e9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:04:44.023507 11564 raft_consensus.cc:2272] T 9cb826017c9347e48bf70204b83b9370 P 49573166d38746089e79505c86c6e93a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:04:44.023514 11565 raft_consensus.cc:2272] T 9cb826017c9347e48bf70204b83b9370 P f49e2dd467f042f08da4345b01c63e9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:04:44.023470 11566 tablet_replica.cc:333] T 9cb826017c9347e48bf70204b83b9370 P 934c8c1fa4f64ba7941168f5ab504ce7: stopping tablet replica
I20260429 14:04:44.023576 11566 raft_consensus.cc:2243] T 9cb826017c9347e48bf70204b83b9370 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 1 LEADER]: Raft consensus shutting down.
I20260429 14:04:44.023695 11566 raft_consensus.cc:2272] T 9cb826017c9347e48bf70204b83b9370 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:04:44.023917 11565 ts_tablet_manager.cc:1916] T 9cb826017c9347e48bf70204b83b9370 P f49e2dd467f042f08da4345b01c63e9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:04:44.024128 11566 ts_tablet_manager.cc:1916] T 9cb826017c9347e48bf70204b83b9370 P 934c8c1fa4f64ba7941168f5ab504ce7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:04:44.024186 11564 ts_tablet_manager.cc:1916] T 9cb826017c9347e48bf70204b83b9370 P 49573166d38746089e79505c86c6e93a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:04:44.025656 11566 ts_tablet_manager.cc:1929] T 9cb826017c9347e48bf70204b83b9370 P 934c8c1fa4f64ba7941168f5ab504ce7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 14:04:44.025674 11565 ts_tablet_manager.cc:1929] T 9cb826017c9347e48bf70204b83b9370 P f49e2dd467f042f08da4345b01c63e9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 14:04:44.025682 11564 ts_tablet_manager.cc:1929] T 9cb826017c9347e48bf70204b83b9370 P 49573166d38746089e79505c86c6e93a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 14:04:44.025715 11566 log.cc:1199] T 9cb826017c9347e48bf70204b83b9370 P 934c8c1fa4f64ba7941168f5ab504ce7: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-1/wal/wals/9cb826017c9347e48bf70204b83b9370
I20260429 14:04:44.025730 11565 log.cc:1199] T 9cb826017c9347e48bf70204b83b9370 P f49e2dd467f042f08da4345b01c63e9b: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-2/wal/wals/9cb826017c9347e48bf70204b83b9370
I20260429 14:04:44.025732 11564 log.cc:1199] T 9cb826017c9347e48bf70204b83b9370 P 49573166d38746089e79505c86c6e93a: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-0/wal/wals/9cb826017c9347e48bf70204b83b9370
I20260429 14:04:44.025944 11566 ts_tablet_manager.cc:1950] T 9cb826017c9347e48bf70204b83b9370 P 934c8c1fa4f64ba7941168f5ab504ce7: Deleting consensus metadata
I20260429 14:04:44.025957 11565 ts_tablet_manager.cc:1950] T 9cb826017c9347e48bf70204b83b9370 P f49e2dd467f042f08da4345b01c63e9b: Deleting consensus metadata
I20260429 14:04:44.025959 11564 ts_tablet_manager.cc:1950] T 9cb826017c9347e48bf70204b83b9370 P 49573166d38746089e79505c86c6e93a: Deleting consensus metadata
I20260429 14:04:44.026342 11010 catalog_manager.cc:5002] TS 49573166d38746089e79505c86c6e93a (127.2.197.193:42967): tablet 9cb826017c9347e48bf70204b83b9370 (table test-table [id=ae913a4988f34bf1b1760543f5952f80]) successfully deleted
I20260429 14:04:44.026392 11009 catalog_manager.cc:5002] TS 934c8c1fa4f64ba7941168f5ab504ce7 (127.2.197.194:37361): tablet 9cb826017c9347e48bf70204b83b9370 (table test-table [id=ae913a4988f34bf1b1760543f5952f80]) successfully deleted
I20260429 14:04:44.026340 11008 catalog_manager.cc:5002] TS f49e2dd467f042f08da4345b01c63e9b (127.2.197.195:37495): tablet 9cb826017c9347e48bf70204b83b9370 (table test-table [id=ae913a4988f34bf1b1760543f5952f80]) successfully deleted
I20260429 14:04:44.027258 11024 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 14:04:44.027441 11024 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 14:04:44.029095 11305 tablet_service.cc:1511] Processing CreateTablet for tablet fff3d2f34a454600b2e32ee945a39da1 (DEFAULT_TABLE table=test-table [id=9df29103ce2d4d6b8f210e4b00c02c12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:04:44.029105 11163 tablet_service.cc:1511] Processing CreateTablet for tablet fff3d2f34a454600b2e32ee945a39da1 (DEFAULT_TABLE table=test-table [id=9df29103ce2d4d6b8f210e4b00c02c12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:04:44.029223 11305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fff3d2f34a454600b2e32ee945a39da1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:44.029244 11163 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fff3d2f34a454600b2e32ee945a39da1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:44.030122 11541 tablet_bootstrap.cc:492] T fff3d2f34a454600b2e32ee945a39da1 P 49573166d38746089e79505c86c6e93a: Bootstrap starting.
I20260429 14:04:44.030478 11542 tablet_bootstrap.cc:492] T fff3d2f34a454600b2e32ee945a39da1 P 934c8c1fa4f64ba7941168f5ab504ce7: Bootstrap starting.
I20260429 14:04:44.030560 11541 tablet_bootstrap.cc:654] T fff3d2f34a454600b2e32ee945a39da1 P 49573166d38746089e79505c86c6e93a: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:44.030532 11447 tablet_service.cc:1511] Processing CreateTablet for tablet fff3d2f34a454600b2e32ee945a39da1 (DEFAULT_TABLE table=test-table [id=9df29103ce2d4d6b8f210e4b00c02c12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:04:44.030648 11447 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fff3d2f34a454600b2e32ee945a39da1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:44.030786 11542 tablet_bootstrap.cc:654] T fff3d2f34a454600b2e32ee945a39da1 P 934c8c1fa4f64ba7941168f5ab504ce7: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:44.031415 11542 tablet_bootstrap.cc:492] T fff3d2f34a454600b2e32ee945a39da1 P 934c8c1fa4f64ba7941168f5ab504ce7: No bootstrap required, opened a new log
I20260429 14:04:44.031467 11542 ts_tablet_manager.cc:1403] T fff3d2f34a454600b2e32ee945a39da1 P 934c8c1fa4f64ba7941168f5ab504ce7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:44.031497 11541 tablet_bootstrap.cc:492] T fff3d2f34a454600b2e32ee945a39da1 P 49573166d38746089e79505c86c6e93a: No bootstrap required, opened a new log
I20260429 14:04:44.031535 11541 ts_tablet_manager.cc:1403] T fff3d2f34a454600b2e32ee945a39da1 P 49573166d38746089e79505c86c6e93a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260429 14:04:44.031608 11542 raft_consensus.cc:359] T fff3d2f34a454600b2e32ee945a39da1 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:44.031672 11542 raft_consensus.cc:385] T fff3d2f34a454600b2e32ee945a39da1 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:44.031641 11541 raft_consensus.cc:359] T fff3d2f34a454600b2e32ee945a39da1 P 49573166d38746089e79505c86c6e93a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:44.031693 11542 raft_consensus.cc:740] T fff3d2f34a454600b2e32ee945a39da1 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 934c8c1fa4f64ba7941168f5ab504ce7, State: Initialized, Role: FOLLOWER
I20260429 14:04:44.031697 11541 raft_consensus.cc:385] T fff3d2f34a454600b2e32ee945a39da1 P 49573166d38746089e79505c86c6e93a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:44.031718 11541 raft_consensus.cc:740] T fff3d2f34a454600b2e32ee945a39da1 P 49573166d38746089e79505c86c6e93a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49573166d38746089e79505c86c6e93a, State: Initialized, Role: FOLLOWER
I20260429 14:04:44.031741 11542 consensus_queue.cc:260] T fff3d2f34a454600b2e32ee945a39da1 P 934c8c1fa4f64ba7941168f5ab504ce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:44.031771 11541 consensus_queue.cc:260] T fff3d2f34a454600b2e32ee945a39da1 P 49573166d38746089e79505c86c6e93a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:44.031836 11542 ts_tablet_manager.cc:1434] T fff3d2f34a454600b2e32ee945a39da1 P 934c8c1fa4f64ba7941168f5ab504ce7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260429 14:04:44.031855 11541 ts_tablet_manager.cc:1434] T fff3d2f34a454600b2e32ee945a39da1 P 49573166d38746089e79505c86c6e93a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:44.031955 11544 tablet_bootstrap.cc:492] T fff3d2f34a454600b2e32ee945a39da1 P f49e2dd467f042f08da4345b01c63e9b: Bootstrap starting.
I20260429 14:04:44.032306 11544 tablet_bootstrap.cc:654] T fff3d2f34a454600b2e32ee945a39da1 P f49e2dd467f042f08da4345b01c63e9b: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:44.033005 11544 tablet_bootstrap.cc:492] T fff3d2f34a454600b2e32ee945a39da1 P f49e2dd467f042f08da4345b01c63e9b: No bootstrap required, opened a new log
I20260429 14:04:44.033069 11544 ts_tablet_manager.cc:1403] T fff3d2f34a454600b2e32ee945a39da1 P f49e2dd467f042f08da4345b01c63e9b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:44.033177 11544 raft_consensus.cc:359] T fff3d2f34a454600b2e32ee945a39da1 P f49e2dd467f042f08da4345b01c63e9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:44.033257 11544 raft_consensus.cc:385] T fff3d2f34a454600b2e32ee945a39da1 P f49e2dd467f042f08da4345b01c63e9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:44.033281 11544 raft_consensus.cc:740] T fff3d2f34a454600b2e32ee945a39da1 P f49e2dd467f042f08da4345b01c63e9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f49e2dd467f042f08da4345b01c63e9b, State: Initialized, Role: FOLLOWER
I20260429 14:04:44.033334 11544 consensus_queue.cc:260] T fff3d2f34a454600b2e32ee945a39da1 P f49e2dd467f042f08da4345b01c63e9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:44.033425 11544 ts_tablet_manager.cc:1434] T fff3d2f34a454600b2e32ee945a39da1 P f49e2dd467f042f08da4345b01c63e9b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:44.087590 11551 raft_consensus.cc:493] T fff3d2f34a454600b2e32ee945a39da1 P f49e2dd467f042f08da4345b01c63e9b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:04:44.087716 11551 raft_consensus.cc:515] T fff3d2f34a454600b2e32ee945a39da1 P f49e2dd467f042f08da4345b01c63e9b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:44.087947 11551 leader_election.cc:290] T fff3d2f34a454600b2e32ee945a39da1 P f49e2dd467f042f08da4345b01c63e9b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49573166d38746089e79505c86c6e93a (127.2.197.193:42967), 934c8c1fa4f64ba7941168f5ab504ce7 (127.2.197.194:37361)
I20260429 14:04:44.088176 11325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fff3d2f34a454600b2e32ee945a39da1" candidate_uuid: "f49e2dd467f042f08da4345b01c63e9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" is_pre_election: true
I20260429 14:04:44.088187 11183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fff3d2f34a454600b2e32ee945a39da1" candidate_uuid: "f49e2dd467f042f08da4345b01c63e9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49573166d38746089e79505c86c6e93a" is_pre_election: true
I20260429 14:04:44.088268 11325 raft_consensus.cc:2468] T fff3d2f34a454600b2e32ee945a39da1 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f49e2dd467f042f08da4345b01c63e9b in term 0.
I20260429 14:04:44.088291 11183 raft_consensus.cc:2468] T fff3d2f34a454600b2e32ee945a39da1 P 49573166d38746089e79505c86c6e93a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f49e2dd467f042f08da4345b01c63e9b in term 0.
I20260429 14:04:44.088433 11399 leader_election.cc:304] T fff3d2f34a454600b2e32ee945a39da1 P f49e2dd467f042f08da4345b01c63e9b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 934c8c1fa4f64ba7941168f5ab504ce7, f49e2dd467f042f08da4345b01c63e9b; no voters:
I20260429 14:04:44.088536 11551 raft_consensus.cc:2804] T fff3d2f34a454600b2e32ee945a39da1 P f49e2dd467f042f08da4345b01c63e9b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 14:04:44.088570 11551 raft_consensus.cc:493] T fff3d2f34a454600b2e32ee945a39da1 P f49e2dd467f042f08da4345b01c63e9b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 14:04:44.088601 11551 raft_consensus.cc:3060] T fff3d2f34a454600b2e32ee945a39da1 P f49e2dd467f042f08da4345b01c63e9b [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:44.089327 11551 raft_consensus.cc:515] T fff3d2f34a454600b2e32ee945a39da1 P f49e2dd467f042f08da4345b01c63e9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:44.089473 11551 leader_election.cc:290] T fff3d2f34a454600b2e32ee945a39da1 P f49e2dd467f042f08da4345b01c63e9b [CANDIDATE]: Term 1 election: Requested vote from peers 49573166d38746089e79505c86c6e93a (127.2.197.193:42967), 934c8c1fa4f64ba7941168f5ab504ce7 (127.2.197.194:37361)
I20260429 14:04:44.089612 11183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fff3d2f34a454600b2e32ee945a39da1" candidate_uuid: "f49e2dd467f042f08da4345b01c63e9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49573166d38746089e79505c86c6e93a"
I20260429 14:04:44.089614 11325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fff3d2f34a454600b2e32ee945a39da1" candidate_uuid: "f49e2dd467f042f08da4345b01c63e9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "934c8c1fa4f64ba7941168f5ab504ce7"
I20260429 14:04:44.089684 11325 raft_consensus.cc:3060] T fff3d2f34a454600b2e32ee945a39da1 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:44.089684 11183 raft_consensus.cc:3060] T fff3d2f34a454600b2e32ee945a39da1 P 49573166d38746089e79505c86c6e93a [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:44.090435 11325 raft_consensus.cc:2468] T fff3d2f34a454600b2e32ee945a39da1 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f49e2dd467f042f08da4345b01c63e9b in term 1.
I20260429 14:04:44.090574 11183 raft_consensus.cc:2468] T fff3d2f34a454600b2e32ee945a39da1 P 49573166d38746089e79505c86c6e93a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f49e2dd467f042f08da4345b01c63e9b in term 1.
I20260429 14:04:44.090613 11399 leader_election.cc:304] T fff3d2f34a454600b2e32ee945a39da1 P f49e2dd467f042f08da4345b01c63e9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 934c8c1fa4f64ba7941168f5ab504ce7, f49e2dd467f042f08da4345b01c63e9b; no voters:
I20260429 14:04:44.090693 11551 raft_consensus.cc:2804] T fff3d2f34a454600b2e32ee945a39da1 P f49e2dd467f042f08da4345b01c63e9b [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:04:44.090844 11551 raft_consensus.cc:697] T fff3d2f34a454600b2e32ee945a39da1 P f49e2dd467f042f08da4345b01c63e9b [term 1 LEADER]: Becoming Leader. State: Replica: f49e2dd467f042f08da4345b01c63e9b, State: Running, Role: LEADER
I20260429 14:04:44.090911 11551 consensus_queue.cc:237] T fff3d2f34a454600b2e32ee945a39da1 P f49e2dd467f042f08da4345b01c63e9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } }
I20260429 14:04:44.091513 11023 catalog_manager.cc:5671] T fff3d2f34a454600b2e32ee945a39da1 P f49e2dd467f042f08da4345b01c63e9b reported cstate change: term changed from 0 to 1, leader changed from <none> to f49e2dd467f042f08da4345b01c63e9b (127.2.197.195). New cstate: current_term: 1 leader_uuid: "f49e2dd467f042f08da4345b01c63e9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f49e2dd467f042f08da4345b01c63e9b" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 37495 } health_report { overall_health: HEALTHY } } }
I20260429 14:04:44.111724 11539 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:44.109369 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37522 (local address 127.2.197.195:37495)
0429 14:04:44.109415 (+ 46us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:44.109417 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:44.109427 (+ 10us) server_negotiation.cc:408] Connection header received
0429 14:04:44.109452 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:44.109453 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:44.109463 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:44.109544 (+ 81us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:04:44.109892 (+ 348us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:44.110418 (+ 526us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:44.111236 (+ 818us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:44.111361 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:44.111378 (+ 17us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:04:44.111398 (+ 20us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:04:44.111453 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:44.111567 (+ 114us) server_negotiation.cc:300] Negotiation successful
0429 14:04:44.111590 (+ 23us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260429 14:04:44.116552 11630 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:44.114131 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:37361 (local address 127.2.197.195:49799)
0429 14:04:44.114289 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:44.114301 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:44.114391 (+ 90us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:44.114594 (+ 203us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:44.114595 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:44.114598 (+ 3us) client_negotiation.cc:190] Negotiated authn=TOKEN
0429 14:04:44.114837 (+ 239us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:44.114839 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:44.115489 (+ 650us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:44.115490 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:44.116002 (+ 512us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:44.116005 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:44.116089 (+ 84us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:44.116103 (+ 14us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0429 14:04:44.116362 (+ 259us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0429 14:04:44.116365 (+ 3us) client_negotiation.cc:770] Sending connection context
0429 14:04:44.116437 (+ 72us) client_negotiation.cc:241] Negotiation successful
0429 14:04:44.116483 (+ 46us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":71,"threads_started":1}
I20260429 14:04:44.116549 11540 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:44.114230 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:49799 (local address 127.2.197.194:37361)
0429 14:04:44.114298 (+ 68us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:44.114301 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:44.114374 (+ 73us) server_negotiation.cc:408] Connection header received
0429 14:04:44.114458 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:44.114459 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:44.114469 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:44.114593 (+ 124us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:04:44.114920 (+ 327us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:44.115412 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:44.116085 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:44.116251 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:44.116269 (+ 18us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:04:44.116290 (+ 21us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:04:44.116359 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:44.116439 (+ 80us) server_negotiation.cc:300] Negotiation successful
0429 14:04:44.116463 (+ 24us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260429 14:04:44.118741 11325 raft_consensus.cc:1275] T fff3d2f34a454600b2e32ee945a39da1 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 1 FOLLOWER]: Refusing update from remote peer f49e2dd467f042f08da4345b01c63e9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:04:44.118954 11551 consensus_queue.cc:1048] T fff3d2f34a454600b2e32ee945a39da1 P f49e2dd467f042f08da4345b01c63e9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "934c8c1fa4f64ba7941168f5ab504ce7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 37361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:04:44.119194 11183 raft_consensus.cc:1275] T fff3d2f34a454600b2e32ee945a39da1 P 49573166d38746089e79505c86c6e93a [term 1 FOLLOWER]: Refusing update from remote peer f49e2dd467f042f08da4345b01c63e9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:04:44.119385 11551 consensus_queue.cc:1048] T fff3d2f34a454600b2e32ee945a39da1 P f49e2dd467f042f08da4345b01c63e9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "49573166d38746089e79505c86c6e93a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:04:44.121042 11555 mvcc.cc:204] Tried to move back new op lower bound from 7280523198948904960 to 7280523198836719616. Current Snapshot: MvccSnapshot[applied={T|T < 7280523198948904960}]
I20260429 14:04:44.127549 11600 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:44.124824 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:37495 (local address 127.2.197.194:38631)
0429 14:04:44.124876 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:44.124888 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:44.124928 (+ 40us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:44.125117 (+ 189us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:44.125118 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:44.125120 (+ 2us) client_negotiation.cc:190] Negotiated authn=TOKEN
0429 14:04:44.125274 (+ 154us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:44.125275 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:44.126229 (+ 954us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:44.126230 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:44.126946 (+ 716us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:44.126949 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:44.127064 (+ 115us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:44.127071 (+ 7us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0429 14:04:44.127360 (+ 289us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0429 14:04:44.127362 (+ 2us) client_negotiation.cc:770] Sending connection context
0429 14:04:44.127423 (+ 61us) client_negotiation.cc:241] Negotiation successful
0429 14:04:44.127485 (+ 62us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260429 14:04:44.127527 11539 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:44.124946 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:38631 (local address 127.2.197.195:37495)
0429 14:04:44.124988 (+ 42us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:44.124990 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:44.124998 (+ 8us) server_negotiation.cc:408] Connection header received
0429 14:04:44.125022 (+ 24us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:44.125023 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:44.125033 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:44.125112 (+ 79us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:04:44.125351 (+ 239us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:44.126113 (+ 762us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:44.127060 (+ 947us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:44.127223 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:44.127242 (+ 19us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:04:44.127264 (+ 22us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:04:44.127355 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:44.127426 (+ 71us) server_negotiation.cc:300] Negotiation successful
0429 14:04:44.127452 (+ 26us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260429 14:04:44.137480 11023 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:04:44.137591 11023 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:04:44.138315 11023 catalog_manager.cc:5958] T 00000000000000000000000000000000 P bb7ccf4eb15e4d9e84fe3634fa4b154b: Sending DeleteTablet for 3 replicas of tablet fff3d2f34a454600b2e32ee945a39da1
I20260429 14:04:44.138566 11163 tablet_service.cc:1558] Processing DeleteTablet for tablet fff3d2f34a454600b2e32ee945a39da1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:04:44 UTC) from {username='oryx', principal='oryx/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:39604
I20260429 14:04:44.138669 11564 tablet_replica.cc:333] T fff3d2f34a454600b2e32ee945a39da1 P 49573166d38746089e79505c86c6e93a: stopping tablet replica
I20260429 14:04:44.138722 11564 raft_consensus.cc:2243] T fff3d2f34a454600b2e32ee945a39da1 P 49573166d38746089e79505c86c6e93a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:04:44.138725 11447 tablet_service.cc:1558] Processing DeleteTablet for tablet fff3d2f34a454600b2e32ee945a39da1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:04:44 UTC) from {username='oryx', principal='oryx/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:37490
I20260429 14:04:44.138773 11564 raft_consensus.cc:2272] T fff3d2f34a454600b2e32ee945a39da1 P 49573166d38746089e79505c86c6e93a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:04:44.138808 11565 tablet_replica.cc:333] T fff3d2f34a454600b2e32ee945a39da1 P f49e2dd467f042f08da4345b01c63e9b: stopping tablet replica
I20260429 14:04:44.138859 11565 raft_consensus.cc:2243] T fff3d2f34a454600b2e32ee945a39da1 P f49e2dd467f042f08da4345b01c63e9b [term 1 LEADER]: Raft consensus shutting down.
I20260429 14:04:44.138955 11565 raft_consensus.cc:2272] T fff3d2f34a454600b2e32ee945a39da1 P f49e2dd467f042f08da4345b01c63e9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:04:44.139110 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 11098
I20260429 14:04:44.140396 11305 tablet_service.cc:1558] Processing DeleteTablet for tablet fff3d2f34a454600b2e32ee945a39da1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:04:44 UTC) from {username='oryx', principal='oryx/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:38694
I20260429 14:04:44.140518 11566 tablet_replica.cc:333] T fff3d2f34a454600b2e32ee945a39da1 P 934c8c1fa4f64ba7941168f5ab504ce7: stopping tablet replica
I20260429 14:04:44.140573 11566 raft_consensus.cc:2243] T fff3d2f34a454600b2e32ee945a39da1 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:04:44.140626 11566 raft_consensus.cc:2272] T fff3d2f34a454600b2e32ee945a39da1 P 934c8c1fa4f64ba7941168f5ab504ce7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:04:44.140956 11566 ts_tablet_manager.cc:1916] T fff3d2f34a454600b2e32ee945a39da1 P 934c8c1fa4f64ba7941168f5ab504ce7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:04:44.141016 11565 ts_tablet_manager.cc:1916] T fff3d2f34a454600b2e32ee945a39da1 P f49e2dd467f042f08da4345b01c63e9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:04:44.142779 11565 ts_tablet_manager.cc:1929] T fff3d2f34a454600b2e32ee945a39da1 P f49e2dd467f042f08da4345b01c63e9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 14:04:44.142822 11566 ts_tablet_manager.cc:1929] T fff3d2f34a454600b2e32ee945a39da1 P 934c8c1fa4f64ba7941168f5ab504ce7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 14:04:44.142845 11565 log.cc:1199] T fff3d2f34a454600b2e32ee945a39da1 P f49e2dd467f042f08da4345b01c63e9b: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-2/wal/wals/fff3d2f34a454600b2e32ee945a39da1
I20260429 14:04:44.142874 11566 log.cc:1199] T fff3d2f34a454600b2e32ee945a39da1 P 934c8c1fa4f64ba7941168f5ab504ce7: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777471413505459-2839-0/minicluster-data/ts-1/wal/wals/fff3d2f34a454600b2e32ee945a39da1
I20260429 14:04:44.143085 11566 ts_tablet_manager.cc:1950] T fff3d2f34a454600b2e32ee945a39da1 P 934c8c1fa4f64ba7941168f5ab504ce7: Deleting consensus metadata
I20260429 14:04:44.143092 11565 ts_tablet_manager.cc:1950] T fff3d2f34a454600b2e32ee945a39da1 P f49e2dd467f042f08da4345b01c63e9b: Deleting consensus metadata
I20260429 14:04:44.143435 11008 catalog_manager.cc:5002] TS f49e2dd467f042f08da4345b01c63e9b (127.2.197.195:37495): tablet fff3d2f34a454600b2e32ee945a39da1 (table test-table [id=9df29103ce2d4d6b8f210e4b00c02c12]) successfully deleted
I20260429 14:04:44.143544 11009 catalog_manager.cc:5002] TS 934c8c1fa4f64ba7941168f5ab504ce7 (127.2.197.194:37361): tablet fff3d2f34a454600b2e32ee945a39da1 (table test-table [id=9df29103ce2d4d6b8f210e4b00c02c12]) successfully deleted
W20260429 14:04:44.148101 11010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.197.193:42967 (error 108)
I20260429 14:04:44.148743 11636 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:44.148512 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:42967 (local address 127.0.0.1:39622)
0429 14:04:44.148664 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:44.148702 (+ 38us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.193:42967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":54,"threads_started":1}
I20260429 14:04:44.148865 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 11240
W20260429 14:04:44.150718 11010 catalog_manager.cc:4729] TS 49573166d38746089e79505c86c6e93a (127.2.197.193:42967): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fff3d2f34a454600b2e32ee945a39da1: Network error: Client connection negotiation failed: client connection to 127.2.197.193:42967: connect: Connection refused (error 111)
I20260429 14:04:44.155727 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 11382
I20260429 14:04:44.162668 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 10992
2026-04-29T14:04:44Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (4043 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:04:44 dist-test-slave-8n77 krb5kdc[11639](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:04:44 dist-test-slave-8n77 krb5kdc[11639](info): set up 2 sockets
Apr 29 14:04:44 dist-test-slave-8n77 krb5kdc[11639](info): commencing operation
krb5kdc: starting...
W20260429 14:04:46.195976 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.011s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:04:46 dist-test-slave-8n77 krb5kdc[11639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:04:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:04:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:04:46.338915 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:33983
--webserver_interface=127.2.197.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:44301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:33983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:04:46.414223 11655 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:46.414474 11655 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:46.414515 11655 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:46.416054 11655 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:04:46.416146 11655 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:46.416170 11655 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:46.416193 11655 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:04:46.416229 11655 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:04:46.417871 11655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:44301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:33983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:33983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:46.418144 11655 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:46.418355 11655 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:46.420889 11660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:46.420850 11663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:46.420866 11661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:46.420954 11655 server_base.cc:1061] running on GCE node
I20260429 14:04:46.421295 11655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:46.421531 11655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:46.422667 11655 hybrid_clock.cc:648] HybridClock initialized: now 1777471486422648 us; error 31 us; skew 500 ppm
W20260429 14:04:46.424366 11655 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-04-29T14:04:46Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2261 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:04:46 dist-test-slave-8n77 krb5kdc[11668](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:04:46 dist-test-slave-8n77 krb5kdc[11668](info): set up 2 sockets
Apr 29 14:04:46 dist-test-slave-8n77 krb5kdc[11668](info): commencing operation
krb5kdc: starting...
W20260429 14:04:48.464373 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.018s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:04:48 dist-test-slave-8n77 krb5kdc[11668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:04:48Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:04:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:04:48.618103 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:37333
--webserver_interface=127.2.197.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:40337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:37333
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:04:48.695948 11684 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:48.696197 11684 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:48.696236 11684 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:48.697790 11684 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:04:48.697886 11684 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:48.697911 11684 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:48.697930 11684 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:04:48.697959 11684 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:04:48.699704 11684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:40337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:37333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:37333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:48.700004 11684 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:48.700198 11684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:48.702857 11690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:48.702841 11692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:48.702841 11689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:48.703635 11684 server_base.cc:1061] running on GCE node
I20260429 14:04:48.703857 11684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:48.704073 11684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:48.705240 11684 hybrid_clock.cc:648] HybridClock initialized: now 1777471488705199 us; error 44 us; skew 500 ppm
Apr 29 14:04:48 dist-test-slave-8n77 krb5kdc[11668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471488, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:48.708069 11684 init.cc:377] Logged in from keytab as kudu/127.2.197.254@KRBTEST.COM (short username kudu)
I20260429 14:04:48.708655 11684 webserver.cc:492] Webserver started at http://127.2.197.254:38375/ using document root <none> and password file <none>
I20260429 14:04:48.708815 11684 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:48.708853 11684 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:48.708925 11684 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:04:48.709816 11684 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "2b78e6df63fb4aaa8eebf66688d45c11"
format_stamp: "Formatted at 2026-04-29 14:04:48 on dist-test-slave-8n77"
server_key: "7deade67f27c038bcb147d449bf758e0"
server_key_iv: "68ed3fed2a0042ea1a9ade1a222a0404"
server_key_version: "encryptionkey@0"
I20260429 14:04:48.710093 11684 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "2b78e6df63fb4aaa8eebf66688d45c11"
format_stamp: "Formatted at 2026-04-29 14:04:48 on dist-test-slave-8n77"
server_key: "7deade67f27c038bcb147d449bf758e0"
server_key_iv: "68ed3fed2a0042ea1a9ade1a222a0404"
server_key_version: "encryptionkey@0"
I20260429 14:04:48.711335 11684 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260429 14:04:48.712260 11699 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:48.712605 11684 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:48.712692 11684 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "2b78e6df63fb4aaa8eebf66688d45c11"
format_stamp: "Formatted at 2026-04-29 14:04:48 on dist-test-slave-8n77"
server_key: "7deade67f27c038bcb147d449bf758e0"
server_key_iv: "68ed3fed2a0042ea1a9ade1a222a0404"
server_key_version: "encryptionkey@0"
I20260429 14:04:48.712885 11684 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:48.742785 11684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:48.745154 11684 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:48.745342 11684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:48.750240 11684 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:37333
I20260429 14:04:48.750435 11751 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:37333 every 8 connection(s)
I20260429 14:04:48.750694 11684 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:04:48.751319 11752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:04:48.753374 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 11684
I20260429 14:04:48.753496 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:04:48.753700 2839 external_mini_cluster.cc:1468] Setting key 57c0f44dd85629a1e13e576eb1dd72ca
I20260429 14:04:48.754386 11752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11: Bootstrap starting.
I20260429 14:04:48.755028 11752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11: Neither blocks nor log segments found. Creating new log.
I20260429 14:04:48.755298 11752 log.cc:826] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11: Log is configured to *not* fsync() on all Append() calls
I20260429 14:04:48.755993 11752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11: No bootstrap required, opened a new log
I20260429 14:04:48.757543 11752 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b78e6df63fb4aaa8eebf66688d45c11" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 37333 } }
I20260429 14:04:48.757754 11752 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:04:48.757790 11752 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b78e6df63fb4aaa8eebf66688d45c11, State: Initialized, Role: FOLLOWER
I20260429 14:04:48.757927 11752 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b78e6df63fb4aaa8eebf66688d45c11" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 37333 } }
I20260429 14:04:48.758038 11752 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:04:48.758083 11752 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:04:48.758155 11752 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:04:48.758970 11752 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b78e6df63fb4aaa8eebf66688d45c11" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 37333 } }
I20260429 14:04:48.759147 11752 leader_election.cc:304] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b78e6df63fb4aaa8eebf66688d45c11; no voters:
I20260429 14:04:48.759332 11752 leader_election.cc:290] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:04:48.759393 11757 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:04:48.759568 11757 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11 [term 1 LEADER]: Becoming Leader. State: Replica: 2b78e6df63fb4aaa8eebf66688d45c11, State: Running, Role: LEADER
I20260429 14:04:48.759593 11752 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:04:48.759718 11757 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b78e6df63fb4aaa8eebf66688d45c11" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 37333 } }
I20260429 14:04:48.760205 11759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b78e6df63fb4aaa8eebf66688d45c11. Latest consensus state: current_term: 1 leader_uuid: "2b78e6df63fb4aaa8eebf66688d45c11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b78e6df63fb4aaa8eebf66688d45c11" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 37333 } } }
I20260429 14:04:48.760800 11759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11 [sys.catalog]: This master's current role is: LEADER
Apr 29 14:04:48 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:04:48.760793 11758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b78e6df63fb4aaa8eebf66688d45c11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b78e6df63fb4aaa8eebf66688d45c11" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 37333 } } }
I20260429 14:04:48.760993 11758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11 [sys.catalog]: This master's current role is: LEADER
I20260429 14:04:48.761111 11766 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:04:48.761607 11766 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:04:48.763567 11766 catalog_manager.cc:1357] Generated new cluster ID: e8b8be05fbd34f42b765dc99a6c1fb31
I20260429 14:04:48.763645 11766 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:04:48.765369 11755 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:48.754320 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33604 (local address 127.2.197.254:37333)
0429 14:04:48.754492 (+ 172us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:48.754504 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:48.754530 (+ 26us) server_negotiation.cc:408] Connection header received
0429 14:04:48.755085 (+ 555us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:48.755101 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:48.755261 (+ 160us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:48.755633 (+ 372us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:48.756581 (+ 948us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:48.757844 (+ 1263us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:48.758645 (+ 801us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:48.758936 (+ 291us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:48.761570 (+ 2634us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:48.761580 (+ 10us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:48.761580 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:48.761592 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:48.763610 (+ 2018us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:48.764022 (+ 412us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:48.764025 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:48.764025 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:48.764079 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:48.764250 (+ 171us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:48.764251 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:48.764252 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:48.764478 (+ 226us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:48.764617 (+ 139us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:48.765082 (+ 465us) server_negotiation.cc:300] Negotiation successful
0429 14:04:48.765207 (+ 125us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260429 14:04:48.779045 11766 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:04:48.780115 11766 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:04:48.790221 11766 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11: Generated new TSK 0
I20260429 14:04:48.790529 11766 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:04:48.855167 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.193:0
--local_ip_for_outbound_sockets=127.2.197.193
--webserver_interface=127.2.197.193
--webserver_port=0
--tserver_master_addrs=127.2.197.254:37333
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:40337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:04:48.950647 11780 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:48.950822 11780 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:48.950846 11780 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:48.952450 11780 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:48.952514 11780 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:48.952544 11780 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.193
I20260429 14:04:48.954337 11780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:40337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.197.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:37333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.193
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:48.954600 11780 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:48.954814 11780 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:48.957759 11786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:48.957767 11785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:48.957813 11780 server_base.cc:1061] running on GCE node
W20260429 14:04:48.957755 11788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:48.958443 11780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:48.958786 11780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:48.959969 11780 hybrid_clock.cc:648] HybridClock initialized: now 1777471488959940 us; error 41 us; skew 500 ppm
Apr 29 14:04:48 dist-test-slave-8n77 krb5kdc[11668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471488, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:48.962859 11780 init.cc:377] Logged in from keytab as kudu/127.2.197.193@KRBTEST.COM (short username kudu)
I20260429 14:04:48.963431 11780 webserver.cc:492] Webserver started at http://127.2.197.193:36549/ using document root <none> and password file <none>
I20260429 14:04:48.963634 11780 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:48.963676 11780 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:48.963778 11780 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:04:48.964748 11780 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-0/data/instance:
uuid: "5c937986c492441b9c34ab2248df08f7"
format_stamp: "Formatted at 2026-04-29 14:04:48 on dist-test-slave-8n77"
server_key: "576cc185a97d135261b32386d26476ac"
server_key_iv: "74f59d7384c4f4361e094e45e627bdad"
server_key_version: "encryptionkey@0"
I20260429 14:04:48.965083 11780 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance:
uuid: "5c937986c492441b9c34ab2248df08f7"
format_stamp: "Formatted at 2026-04-29 14:04:48 on dist-test-slave-8n77"
server_key: "576cc185a97d135261b32386d26476ac"
server_key_iv: "74f59d7384c4f4361e094e45e627bdad"
server_key_version: "encryptionkey@0"
I20260429 14:04:48.966421 11780 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260429 14:04:48.967424 11795 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:48.967693 11780 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:48.967779 11780 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-0/wal
uuid: "5c937986c492441b9c34ab2248df08f7"
format_stamp: "Formatted at 2026-04-29 14:04:48 on dist-test-slave-8n77"
server_key: "576cc185a97d135261b32386d26476ac"
server_key_iv: "74f59d7384c4f4361e094e45e627bdad"
server_key_version: "encryptionkey@0"
I20260429 14:04:48.967878 11780 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:48.983389 11780 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:48.985451 11780 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:48.985687 11780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:48.985944 11780 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:04:48.986353 11780 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:04:48.986407 11780 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:48.986452 11780 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:04:48.986481 11780 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:48.993223 11780 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.193:41527
I20260429 14:04:48.993263 11910 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.193:41527 every 8 connection(s)
I20260429 14:04:48.993609 11780 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
Apr 29 14:04:48 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471488, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:04:49.001286 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 11780
I20260429 14:04:49.001443 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance
I20260429 14:04:49.001639 2839 external_mini_cluster.cc:1468] Setting key 7d46ebaf835739784b9909acf84e5c86
I20260429 14:04:49.003190 11755 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:48.994759 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:46445 (local address 127.2.197.254:37333)
0429 14:04:48.994884 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:48.994887 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:48.995063 (+ 176us) server_negotiation.cc:408] Connection header received
0429 14:04:48.995598 (+ 535us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:48.995599 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:48.995612 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:48.995654 (+ 42us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:48.996521 (+ 867us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:48.997081 (+ 560us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:48.997722 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:48.997896 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:49.000328 (+ 2432us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:49.000337 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:49.000338 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:49.000344 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:49.001895 (+ 1551us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:49.002320 (+ 425us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:49.002322 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:49.002322 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:49.002348 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:49.002541 (+ 193us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:49.002543 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:49.002543 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:49.002647 (+ 104us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:49.002724 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:49.002980 (+ 256us) server_negotiation.cc:300] Negotiation successful
0429 14:04:49.003097 (+ 117us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260429 14:04:49.003283 11913 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:48.994744 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.193:46445)
0429 14:04:48.994947 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:48.994978 (+ 31us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:48.995490 (+ 512us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:48.995723 (+ 233us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:48.995727 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:48.996169 (+ 442us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:48.996425 (+ 256us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:48.996428 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:48.997234 (+ 806us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:48.997235 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:48.997629 (+ 394us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:48.997631 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:48.997773 (+ 142us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:48.998756 (+ 983us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:48.998764 (+ 8us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:49.000216 (+ 1452us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:49.001990 (+ 1774us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:49.001996 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:49.001999 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:49.002210 (+ 211us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:49.002408 (+ 198us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:49.002408 (+ 0us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:49.002409 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:49.002476 (+ 67us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:49.002714 (+ 238us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:49.002719 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:49.002847 (+ 128us) client_negotiation.cc:770] Sending connection context
0429 14:04:49.002961 (+ 114us) client_negotiation.cc:241] Negotiation successful
0429 14:04:49.003097 (+ 136us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":82,"threads_started":1}
I20260429 14:04:49.003868 11911 heartbeater.cc:344] Connected to a master server at 127.2.197.254:37333
I20260429 14:04:49.003963 11911 heartbeater.cc:461] Registering TS with master...
I20260429 14:04:49.004184 11911 heartbeater.cc:507] Master 127.2.197.254:37333 requested a full tablet report, sending...
I20260429 14:04:49.004755 11716 ts_manager.cc:194] Registered new tserver with Master: 5c937986c492441b9c34ab2248df08f7 (127.2.197.193:41527)
I20260429 14:04:49.005398 11716 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.193@KRBTEST.COM'} at 127.2.197.193:46445
WARNING: no policy specified for kudu/127.2.197.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:04:49.057754 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.194:0
--local_ip_for_outbound_sockets=127.2.197.194
--webserver_interface=127.2.197.194
--webserver_port=0
--tserver_master_addrs=127.2.197.254:37333
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:40337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:04:49.136458 11918 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:49.136654 11918 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:49.136680 11918 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:49.138180 11918 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:49.138230 11918 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:49.138267 11918 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.194
I20260429 14:04:49.139926 11918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:40337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.197.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:37333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.194
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:49.140120 11918 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:49.140257 11918 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260429 14:04:49.142889 11918 server_base.cc:1061] running on GCE node
W20260429 14:04:49.142858 11923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:49.142858 11924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:49.142858 11926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:49.143319 11918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:49.143584 11918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:49.144737 11918 hybrid_clock.cc:648] HybridClock initialized: now 1777471489144716 us; error 52 us; skew 500 ppm
Apr 29 14:04:49 dist-test-slave-8n77 krb5kdc[11668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471489, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:49.147575 11918 init.cc:377] Logged in from keytab as kudu/127.2.197.194@KRBTEST.COM (short username kudu)
I20260429 14:04:49.148155 11918 webserver.cc:492] Webserver started at http://127.2.197.194:35813/ using document root <none> and password file <none>
I20260429 14:04:49.148391 11918 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:49.148451 11918 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:49.148559 11918 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:04:49.149981 11918 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-1/data/instance:
uuid: "e93a45ecb9344c89806a7d0fce98684a"
format_stamp: "Formatted at 2026-04-29 14:04:49 on dist-test-slave-8n77"
server_key: "5b009e4bf9a097ce3fef95e5df952f99"
server_key_iv: "8379d2e2229a47642636e26c5be999e3"
server_key_version: "encryptionkey@0"
I20260429 14:04:49.150409 11918 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance:
uuid: "e93a45ecb9344c89806a7d0fce98684a"
format_stamp: "Formatted at 2026-04-29 14:04:49 on dist-test-slave-8n77"
server_key: "5b009e4bf9a097ce3fef95e5df952f99"
server_key_iv: "8379d2e2229a47642636e26c5be999e3"
server_key_version: "encryptionkey@0"
I20260429 14:04:49.151731 11918 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:49.152478 11933 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:49.152725 11918 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:49.152803 11918 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-1/wal
uuid: "e93a45ecb9344c89806a7d0fce98684a"
format_stamp: "Formatted at 2026-04-29 14:04:49 on dist-test-slave-8n77"
server_key: "5b009e4bf9a097ce3fef95e5df952f99"
server_key_iv: "8379d2e2229a47642636e26c5be999e3"
server_key_version: "encryptionkey@0"
I20260429 14:04:49.152860 11918 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:49.182117 11918 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:49.183981 11918 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:49.184094 11918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:49.184278 11918 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:04:49.184613 11918 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:04:49.184670 11918 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:49.184718 11918 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:04:49.184744 11918 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:49.190583 11918 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.194:38681
I20260429 14:04:49.190649 12047 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.194:38681 every 8 connection(s)
I20260429 14:04:49.190971 11918 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
I20260429 14:04:49.192835 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 11918
I20260429 14:04:49.192927 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance
I20260429 14:04:49.193159 2839 external_mini_cluster.cc:1468] Setting key 712ab461d38abde415c5bfcff5bf05b3
Apr 29 14:04:49 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471489, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:04:49.200575 11755 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:49.191754 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:37319 (local address 127.2.197.254:37333)
0429 14:04:49.191799 (+ 45us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:49.191801 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:49.192055 (+ 254us) server_negotiation.cc:408] Connection header received
0429 14:04:49.192593 (+ 538us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:49.192594 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:49.192606 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:49.192664 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:49.193605 (+ 941us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:49.194175 (+ 570us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:49.194803 (+ 628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:49.194969 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:49.197253 (+ 2284us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:49.197257 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:49.197258 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:49.197265 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:49.199141 (+ 1876us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:49.199524 (+ 383us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:49.199525 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:49.199525 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:49.199551 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:49.199745 (+ 194us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:49.199746 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:49.199747 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:49.199869 (+ 122us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:49.199941 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:49.200380 (+ 439us) server_negotiation.cc:300] Negotiation successful
0429 14:04:49.200473 (+ 93us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260429 14:04:49.200667 12050 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:49.191726 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.194:37319)
0429 14:04:49.191863 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:49.191959 (+ 96us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:49.192460 (+ 501us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:49.192737 (+ 277us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:49.192741 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:49.193139 (+ 398us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:49.193521 (+ 382us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:49.193524 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:49.194301 (+ 777us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:49.194302 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:49.194703 (+ 401us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:49.194705 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:49.194844 (+ 139us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:49.195769 (+ 925us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:49.195778 (+ 9us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:49.197158 (+ 1380us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:49.199234 (+ 2076us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:49.199239 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:49.199241 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:49.199443 (+ 202us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:49.199619 (+ 176us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:49.199620 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:49.199620 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:49.199673 (+ 53us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:49.200100 (+ 427us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:49.200105 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:49.200231 (+ 126us) client_negotiation.cc:770] Sending connection context
0429 14:04:49.200332 (+ 101us) client_negotiation.cc:241] Negotiation successful
0429 14:04:49.200448 (+ 116us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":76,"threads_started":1}
I20260429 14:04:49.201221 12048 heartbeater.cc:344] Connected to a master server at 127.2.197.254:37333
I20260429 14:04:49.201311 12048 heartbeater.cc:461] Registering TS with master...
I20260429 14:04:49.201548 12048 heartbeater.cc:507] Master 127.2.197.254:37333 requested a full tablet report, sending...
I20260429 14:04:49.201987 11716 ts_manager.cc:194] Registered new tserver with Master: e93a45ecb9344c89806a7d0fce98684a (127.2.197.194:38681)
WARNING: no policy specified for kudu/127.2.197.195@KRBTEST.COM; defaulting to no policy
I20260429 14:04:49.202533 11716 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.194@KRBTEST.COM'} at 127.2.197.194:37319
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:04:49.246160 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.195:0
--local_ip_for_outbound_sockets=127.2.197.195
--webserver_interface=127.2.197.195
--webserver_port=0
--tserver_master_addrs=127.2.197.254:37333
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:40337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:04:49.324090 12055 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:49.324298 12055 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:49.324334 12055 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:49.325814 12055 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:49.325894 12055 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:49.325937 12055 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.195
I20260429 14:04:49.327664 12055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:40337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.197.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:37333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.195
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:49.327927 12055 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:49.328117 12055 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:49.330646 12061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:49.330636 12060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:49.330654 12063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:49.330760 12055 server_base.cc:1061] running on GCE node
I20260429 14:04:49.331019 12055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:49.331246 12055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:49.332414 12055 hybrid_clock.cc:648] HybridClock initialized: now 1777471489332355 us; error 77 us; skew 500 ppm
Apr 29 14:04:49 dist-test-slave-8n77 krb5kdc[11668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471489, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:49.334941 12055 init.cc:377] Logged in from keytab as kudu/127.2.197.195@KRBTEST.COM (short username kudu)
I20260429 14:04:49.335438 12055 webserver.cc:492] Webserver started at http://127.2.197.195:36209/ using document root <none> and password file <none>
I20260429 14:04:49.335614 12055 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:49.335652 12055 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:49.335737 12055 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:04:49.336606 12055 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-2/data/instance:
uuid: "83d3d17ea64a49288f23ae9a20f84b87"
format_stamp: "Formatted at 2026-04-29 14:04:49 on dist-test-slave-8n77"
server_key: "cd3ae26c428c5481692b2e94ccf2a120"
server_key_iv: "4cbebfba63a3dcd36ddf4fafd2ab9da8"
server_key_version: "encryptionkey@0"
I20260429 14:04:49.336879 12055 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance:
uuid: "83d3d17ea64a49288f23ae9a20f84b87"
format_stamp: "Formatted at 2026-04-29 14:04:49 on dist-test-slave-8n77"
server_key: "cd3ae26c428c5481692b2e94ccf2a120"
server_key_iv: "4cbebfba63a3dcd36ddf4fafd2ab9da8"
server_key_version: "encryptionkey@0"
I20260429 14:04:49.338006 12055 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260429 14:04:49.338754 12070 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:49.338958 12055 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260429 14:04:49.339026 12055 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-2/wal
uuid: "83d3d17ea64a49288f23ae9a20f84b87"
format_stamp: "Formatted at 2026-04-29 14:04:49 on dist-test-slave-8n77"
server_key: "cd3ae26c428c5481692b2e94ccf2a120"
server_key_iv: "4cbebfba63a3dcd36ddf4fafd2ab9da8"
server_key_version: "encryptionkey@0"
I20260429 14:04:49.339072 12055 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:49.354430 12055 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:49.356467 12055 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:49.356629 12055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:49.356863 12055 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:04:49.357194 12055 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:04:49.357249 12055 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:49.357293 12055 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:04:49.357318 12055 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:49.363547 12055 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.195:42923
I20260429 14:04:49.363662 12184 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.195:42923 every 8 connection(s)
I20260429 14:04:49.363948 12055 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
Apr 29 14:04:49 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471489, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:04:49.371132 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 12055
I20260429 14:04:49.371233 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance
I20260429 14:04:49.371420 2839 external_mini_cluster.cc:1468] Setting key e710c84668a67eab430104bee6d88b0a
I20260429 14:04:49.372855 11755 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:49.364775 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:53067 (local address 127.2.197.254:37333)
0429 14:04:49.364913 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:49.364915 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:49.365115 (+ 200us) server_negotiation.cc:408] Connection header received
0429 14:04:49.365669 (+ 554us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:49.365671 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:49.365683 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:49.365747 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:49.366507 (+ 760us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:49.366995 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:49.367597 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:49.367763 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:49.370016 (+ 2253us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:49.370020 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:49.370021 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:49.370029 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:49.371462 (+ 1433us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:49.371969 (+ 507us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:49.371970 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:49.371971 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:49.371999 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:49.372219 (+ 220us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:49.372220 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:49.372221 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:49.372359 (+ 138us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:49.372426 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:49.372678 (+ 252us) server_negotiation.cc:300] Negotiation successful
0429 14:04:49.372768 (+ 90us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260429 14:04:49.372980 12187 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:49.364736 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.195:53067)
0429 14:04:49.364918 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:49.364957 (+ 39us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:49.365531 (+ 574us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:49.365806 (+ 275us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:49.365811 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:49.366121 (+ 310us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:49.366403 (+ 282us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:49.366406 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:49.367099 (+ 693us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:49.367100 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:49.367494 (+ 394us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:49.367495 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:49.367641 (+ 146us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:49.368534 (+ 893us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:49.368543 (+ 9us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:49.369922 (+ 1379us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:04:49.371575 (+ 1653us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:49.371580 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:49.371582 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:49.371877 (+ 295us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:49.372068 (+ 191us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:04:49.372069 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:04:49.372069 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:04:49.372142 (+ 73us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:04:49.372414 (+ 272us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:04:49.372418 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:04:49.372545 (+ 127us) client_negotiation.cc:770] Sending connection context
0429 14:04:49.372657 (+ 112us) client_negotiation.cc:241] Negotiation successful
0429 14:04:49.372791 (+ 134us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":80,"threads_started":1}
I20260429 14:04:49.373468 12185 heartbeater.cc:344] Connected to a master server at 127.2.197.254:37333
I20260429 14:04:49.373544 12185 heartbeater.cc:461] Registering TS with master...
I20260429 14:04:49.373698 12185 heartbeater.cc:507] Master 127.2.197.254:37333 requested a full tablet report, sending...
I20260429 14:04:49.374027 11716 ts_manager.cc:194] Registered new tserver with Master: 83d3d17ea64a49288f23ae9a20f84b87 (127.2.197.195:42923)
I20260429 14:04:49.374450 11716 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.195@KRBTEST.COM'} at 127.2.197.195:53067
I20260429 14:04:49.383709 2839 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/oryx_127.2.197.193.keytab.
Entry for principal oryx/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/oryx_127.2.197.193.keytab.
WARNING: no policy specified for oryx/127.2.197.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/oryx_127.2.197.194.keytab.
Entry for principal oryx/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/oryx_127.2.197.194.keytab.
WARNING: no policy specified for oryx/127.2.197.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/oryx_127.2.197.195.keytab.
Entry for principal oryx/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/oryx_127.2.197.195.keytab.
I20260429 14:04:49.462447 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 11780
I20260429 14:04:49.467502 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 11918
I20260429 14:04:49.472486 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 12055
I20260429 14:04:49.477355 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 11684
I20260429 14:04:49.482998 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:37333
--webserver_interface=127.2.197.254
--webserver_port=38375
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:40337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:37333
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:04:49.564922 12194 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:49.565166 12194 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:49.565207 12194 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:49.566982 12194 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:04:49.567086 12194 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:49.567112 12194 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:49.567132 12194 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:04:49.567164 12194 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:04:49.568912 12194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:40337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:37333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:37333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=38375
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:49.569188 12194 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:49.569372 12194 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:49.571890 12200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:49.571887 12202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:49.571982 12194 server_base.cc:1061] running on GCE node
W20260429 14:04:49.571928 12199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:49.572360 12194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:49.572605 12194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:49.573765 12194 hybrid_clock.cc:648] HybridClock initialized: now 1777471489573754 us; error 32 us; skew 500 ppm
Apr 29 14:04:49 dist-test-slave-8n77 krb5kdc[11668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471489, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:49.576416 12194 init.cc:377] Logged in from keytab as kudu/127.2.197.254@KRBTEST.COM (short username kudu)
I20260429 14:04:49.576978 12194 webserver.cc:492] Webserver started at http://127.2.197.254:38375/ using document root <none> and password file <none>
I20260429 14:04:49.577193 12194 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:49.577266 12194 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:49.578588 12194 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:04:49.579291 12209 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:49.579468 12194 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:04:49.579591 12194 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "2b78e6df63fb4aaa8eebf66688d45c11"
format_stamp: "Formatted at 2026-04-29 14:04:48 on dist-test-slave-8n77"
server_key: "7deade67f27c038bcb147d449bf758e0"
server_key_iv: "68ed3fed2a0042ea1a9ade1a222a0404"
server_key_version: "encryptionkey@0"
I20260429 14:04:49.579890 12194 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:49.592657 12194 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:49.594728 12194 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:49.594895 12194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:49.598977 12194 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:37333
I20260429 14:04:49.599026 12262 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:37333 every 8 connection(s)
I20260429 14:04:49.599347 12194 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:04:49.600447 12263 sys_catalog.cc:263] Verifying existing consensus state
I20260429 14:04:49.600962 12263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11: Bootstrap starting.
I20260429 14:04:49.607805 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 12194
I20260429 14:04:49.608187 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.193:41527
--local_ip_for_outbound_sockets=127.2.197.193
--tserver_master_addrs=127.2.197.254:37333
--webserver_port=36549
--webserver_interface=127.2.197.193
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:40337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.197.193
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/oryx_127.2.197.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260429 14:04:49.610839 12263 log.cc:826] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11: Log is configured to *not* fsync() on all Append() calls
I20260429 14:04:49.613147 12263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260429 14:04:49.613507 12263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11: Bootstrap complete.
I20260429 14:04:49.617340 12263 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b78e6df63fb4aaa8eebf66688d45c11" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 37333 } }
I20260429 14:04:49.617689 12263 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b78e6df63fb4aaa8eebf66688d45c11, State: Initialized, Role: FOLLOWER
I20260429 14:04:49.617852 12263 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b78e6df63fb4aaa8eebf66688d45c11" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 37333 } }
I20260429 14:04:49.617935 12263 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:04:49.617980 12263 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:04:49.618032 12263 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11 [term 1 FOLLOWER]: Advancing to term 2
I20260429 14:04:49.619341 12263 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b78e6df63fb4aaa8eebf66688d45c11" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 37333 } }
I20260429 14:04:49.619463 12263 leader_election.cc:304] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b78e6df63fb4aaa8eebf66688d45c11; no voters:
I20260429 14:04:49.619674 12263 leader_election.cc:290] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260429 14:04:49.619766 12267 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11 [term 2 FOLLOWER]: Leader election won for term 2
I20260429 14:04:49.619918 12263 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:04:49.620007 12267 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11 [term 2 LEADER]: Becoming Leader. State: Replica: 2b78e6df63fb4aaa8eebf66688d45c11, State: Running, Role: LEADER
I20260429 14:04:49.620153 12267 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b78e6df63fb4aaa8eebf66688d45c11" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 37333 } }
I20260429 14:04:49.620488 12268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2b78e6df63fb4aaa8eebf66688d45c11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b78e6df63fb4aaa8eebf66688d45c11" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 37333 } } }
I20260429 14:04:49.620559 12268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11 [sys.catalog]: This master's current role is: LEADER
I20260429 14:04:49.620826 12269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b78e6df63fb4aaa8eebf66688d45c11. Latest consensus state: current_term: 2 leader_uuid: "2b78e6df63fb4aaa8eebf66688d45c11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b78e6df63fb4aaa8eebf66688d45c11" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 37333 } } }
I20260429 14:04:49.620857 12276 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:04:49.620882 12269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11 [sys.catalog]: This master's current role is: LEADER
I20260429 14:04:49.621325 12276 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:04:49.621517 12276 catalog_manager.cc:1269] Loaded cluster ID: e8b8be05fbd34f42b765dc99a6c1fb31
I20260429 14:04:49.621822 12276 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:04:49.623272 12276 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:04:49.623472 12276 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 2b78e6df63fb4aaa8eebf66688d45c11: Loaded TSK: 0
I20260429 14:04:49.623665 12276 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260429 14:04:49.688974 12265 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:49.689155 12265 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:49.689181 12265 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:49.690804 12265 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:49.690855 12265 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:49.690882 12265 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.193
I20260429 14:04:49.692556 12265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:40337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/oryx_127.2.197.193.keytab
--principal=oryx/127.2.197.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.193:41527
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.197.193
--webserver_port=36549
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:37333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.193
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:49.692785 12265 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:49.692974 12265 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:49.695541 12290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:49.695551 12291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:49.695793 12265 server_base.cc:1061] running on GCE node
W20260429 14:04:49.695562 12293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:49.696152 12265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:49.696394 12265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:49.697582 12265 hybrid_clock.cc:648] HybridClock initialized: now 1777471489697584 us; error 49 us; skew 500 ppm
Apr 29 14:04:49 dist-test-slave-8n77 krb5kdc[11668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471489, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:49.700718 12265 init.cc:377] Logged in from keytab as oryx/127.2.197.193@KRBTEST.COM (short username oryx)
I20260429 14:04:49.701324 12265 webserver.cc:492] Webserver started at http://127.2.197.193:36549/ using document root <none> and password file <none>
I20260429 14:04:49.701526 12265 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:49.701601 12265 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:49.702999 12265 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:49.703687 12300 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:49.703933 12265 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:49.704017 12265 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-0/wal
uuid: "5c937986c492441b9c34ab2248df08f7"
format_stamp: "Formatted at 2026-04-29 14:04:48 on dist-test-slave-8n77"
server_key: "576cc185a97d135261b32386d26476ac"
server_key_iv: "74f59d7384c4f4361e094e45e627bdad"
server_key_version: "encryptionkey@0"
I20260429 14:04:49.704303 12265 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:49.720252 12265 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:49.722755 12265 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:49.722950 12265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:49.723228 12265 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:04:49.723539 12265 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:04:49.723593 12265 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:49.723641 12265 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:04:49.723673 12265 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:49.729339 12265 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.193:41527
I20260429 14:04:49.729440 12414 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.193:41527 every 8 connection(s)
I20260429 14:04:49.729773 12265 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
I20260429 14:04:49.733407 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 12265
I20260429 14:04:49.733736 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.194:38681
--local_ip_for_outbound_sockets=127.2.197.194
--tserver_master_addrs=127.2.197.254:37333
--webserver_port=35813
--webserver_interface=127.2.197.194
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:40337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.197.194
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/oryx_127.2.197.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Apr 29 14:04:49 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:04:49 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:04:49.747246 12417 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:49.730593 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.193:47533)
0429 14:04:49.730790 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:49.730826 (+ 36us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:49.731406 (+ 580us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:49.731798 (+ 392us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:49.731803 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:49.732215 (+ 412us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:49.732595 (+ 380us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:49.732599 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:49.733630 (+ 1031us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:49.733631 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:49.734138 (+ 507us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:49.734140 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:49.734277 (+ 137us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:49.735272 (+ 995us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:49.735282 (+ 10us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:49.747056 (+ 11774us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":106,"threads_started":1}
W20260429 14:04:49.747373 12418 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:49.730721 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:47533 (local address 127.2.197.254:37333)
0429 14:04:49.730882 (+ 161us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:49.730890 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:49.730910 (+ 20us) server_negotiation.cc:408] Connection header received
0429 14:04:49.731587 (+ 677us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:49.731593 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:49.731629 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:49.731724 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:49.732711 (+ 987us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:49.733542 (+ 831us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:49.734226 (+ 684us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:49.734495 (+ 269us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:49.747214 (+ 12719us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.193:47533: BlockingRecv error: recv got EOF from 127.2.197.193:47533 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":92,"threads_started":1}
W20260429 14:04:49.747913 12415 heartbeater.cc:646] Failed to heartbeat to 127.2.197.254:37333 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.197.254:37333: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Apr 29 14:04:49 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:04:49 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:04:49.752518 12417 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:49.748242 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.193:50823)
0429 14:04:49.748343 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:49.748356 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:49.748412 (+ 56us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:49.748565 (+ 153us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:49.748566 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:49.748714 (+ 148us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:49.748895 (+ 181us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:49.748897 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:49.749589 (+ 692us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:49.749590 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:49.749959 (+ 369us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:49.749960 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:49.750041 (+ 81us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:49.750653 (+ 612us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:49.750658 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:49.752431 (+ 1773us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":81}
W20260429 14:04:49.752573 12418 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:49.748296 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:50823 (local address 127.2.197.254:37333)
0429 14:04:49.748350 (+ 54us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:49.748353 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:49.748394 (+ 41us) server_negotiation.cc:408] Connection header received
0429 14:04:49.748477 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:49.748478 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:49.748496 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:49.748550 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:49.748983 (+ 433us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:49.749500 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:49.750050 (+ 550us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:49.750249 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:49.752512 (+ 2263us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.193:50823: BlockingRecv error: recv got EOF from 127.2.197.193:50823 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Apr 29 14:04:49 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:04:49 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:04:49.757730 12417 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:49.753133 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.193:47205)
0429 14:04:49.753209 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:49.753221 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:49.753271 (+ 50us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:49.753405 (+ 134us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:49.753406 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:49.753543 (+ 137us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:49.753671 (+ 128us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:49.753673 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:49.754398 (+ 725us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:49.754399 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:49.754750 (+ 351us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:49.754751 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:49.754862 (+ 111us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:49.755912 (+ 1050us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:49.755917 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:49.757648 (+ 1731us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":63}
W20260429 14:04:49.757807 12418 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:49.753172 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:47205 (local address 127.2.197.254:37333)
0429 14:04:49.753210 (+ 38us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:49.753212 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:49.753255 (+ 43us) server_negotiation.cc:408] Connection header received
0429 14:04:49.753335 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:49.753335 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:49.753346 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:49.753401 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:49.753760 (+ 359us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:49.754243 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:49.754875 (+ 632us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:49.755244 (+ 369us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:49.757740 (+ 2496us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.193:47205: BlockingRecv error: recv got EOF from 127.2.197.193:47205 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260429 14:04:49.758109 12415 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260429 14:04:49.830215 12419 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:49.830467 12419 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:49.830506 12419 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:49.832010 12419 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:49.832085 12419 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:49.832131 12419 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.194
I20260429 14:04:49.834146 12419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:40337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/oryx_127.2.197.194.keytab
--principal=oryx/127.2.197.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.194:38681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.197.194
--webserver_port=35813
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:37333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.194
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:49.834472 12419 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:49.834677 12419 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:49.837225 12424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:49.837225 12425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:49.837239 12427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:49.837440 12419 server_base.cc:1061] running on GCE node
I20260429 14:04:49.837666 12419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:49.837875 12419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:49.839056 12419 hybrid_clock.cc:648] HybridClock initialized: now 1777471489839040 us; error 25 us; skew 500 ppm
Apr 29 14:04:49 dist-test-slave-8n77 krb5kdc[11668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471489, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:49.841431 12419 init.cc:377] Logged in from keytab as oryx/127.2.197.194@KRBTEST.COM (short username oryx)
I20260429 14:04:49.842020 12419 webserver.cc:492] Webserver started at http://127.2.197.194:35813/ using document root <none> and password file <none>
I20260429 14:04:49.842212 12419 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:49.842306 12419 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:49.843747 12419 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:49.844385 12434 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:49.844655 12419 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:04:49.844741 12419 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-1/wal
uuid: "e93a45ecb9344c89806a7d0fce98684a"
format_stamp: "Formatted at 2026-04-29 14:04:49 on dist-test-slave-8n77"
server_key: "5b009e4bf9a097ce3fef95e5df952f99"
server_key_iv: "8379d2e2229a47642636e26c5be999e3"
server_key_version: "encryptionkey@0"
I20260429 14:04:49.845021 12419 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:49.855196 12419 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:49.857142 12419 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:49.857296 12419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:49.857514 12419 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:04:49.857817 12419 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:04:49.857872 12419 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:49.857911 12419 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:04:49.857935 12419 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:49.864327 12419 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.194:38681
I20260429 14:04:49.864398 12548 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.194:38681 every 8 connection(s)
I20260429 14:04:49.864727 12419 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
Apr 29 14:04:49 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:04:49.870165 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 12419
I20260429 14:04:49.870718 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.195:42923
--local_ip_for_outbound_sockets=127.2.197.195
--tserver_master_addrs=127.2.197.254:37333
--webserver_port=36209
--webserver_interface=127.2.197.195
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:40337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.197.195
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/oryx_127.2.197.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Apr 29 14:04:49 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:04:49.871820 12551 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:49.865441 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.194:52023)
0429 14:04:49.865586 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:49.865615 (+ 29us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:49.866097 (+ 482us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:49.866588 (+ 491us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:49.866591 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:49.866906 (+ 315us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:49.867157 (+ 251us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:49.867161 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:49.867878 (+ 717us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:49.867879 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:49.868273 (+ 394us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:49.868275 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:49.868369 (+ 94us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:49.869249 (+ 880us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:49.869259 (+ 10us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:49.871658 (+ 2399us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":53,"threads_started":1}
W20260429 14:04:49.871879 12418 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:49.865530 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:52023 (local address 127.2.197.254:37333)
0429 14:04:49.865615 (+ 85us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:49.865618 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:49.865707 (+ 89us) server_negotiation.cc:408] Connection header received
0429 14:04:49.866258 (+ 551us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:49.866260 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:49.866457 (+ 197us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:49.866518 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:49.867264 (+ 746us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:49.867769 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:49.868351 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:49.868461 (+ 110us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:49.871791 (+ 3330us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.194:52023: BlockingRecv error: recv got EOF from 127.2.197.194:52023 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260429 14:04:49.872316 12549 heartbeater.cc:646] Failed to heartbeat to 127.2.197.254:37333 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.197.254:37333: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Apr 29 14:04:49 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:04:49 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:04:49.877564 12551 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:49.872701 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.194:34547)
0429 14:04:49.872758 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:49.872773 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:49.872819 (+ 46us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:49.872993 (+ 174us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:49.872995 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:49.873113 (+ 118us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:49.873267 (+ 154us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:49.873268 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:49.874305 (+ 1037us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:49.874307 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:49.874976 (+ 669us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:49.874980 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:49.875060 (+ 80us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:49.875559 (+ 499us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:49.875565 (+ 6us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:49.877469 (+ 1904us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260429 14:04:49.877763 12418 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:49.872775 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:34547 (local address 127.2.197.254:37333)
0429 14:04:49.872815 (+ 40us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:49.872817 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:49.872826 (+ 9us) server_negotiation.cc:408] Connection header received
0429 14:04:49.872886 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:49.872887 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:49.872900 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:49.872964 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:49.873347 (+ 383us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:49.874169 (+ 822us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:49.875063 (+ 894us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:49.875344 (+ 281us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:49.877658 (+ 2314us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.194:34547: BlockingRecv error: recv got EOF from 127.2.197.194:34547 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Apr 29 14:04:49 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:04:49 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:04:49.886283 12551 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:49.880165 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.194:33661)
0429 14:04:49.880214 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:49.880224 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:49.880263 (+ 39us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:49.881245 (+ 982us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:49.881246 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:49.881384 (+ 138us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:49.881555 (+ 171us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:49.881565 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:49.883070 (+ 1505us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:49.883071 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:49.883767 (+ 696us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:49.883770 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:49.883851 (+ 81us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:49.884291 (+ 440us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:49.884296 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:49.886192 (+ 1896us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260429 14:04:49.886360 12418 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:49.880279 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:33661 (local address 127.2.197.254:37333)
0429 14:04:49.881087 (+ 808us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:49.881089 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:49.881098 (+ 9us) server_negotiation.cc:408] Connection header received
0429 14:04:49.881141 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:49.881143 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:49.881155 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:49.881211 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:49.882340 (+ 1129us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:49.882972 (+ 632us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:49.883871 (+ 899us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:49.884043 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:49.886298 (+ 2255us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.194:33661: BlockingRecv error: recv got EOF from 127.2.197.194:33661 (error 108)
Metrics: {"server-negotiator.queue_time_us":785}
W20260429 14:04:49.886641 12549 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260429 14:04:49.953038 12552 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:04:49.953253 12552 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:04:49.953284 12552 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:04:49.955116 12552 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:04:49.955183 12552 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:04:49.955214 12552 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.195
I20260429 14:04:49.956876 12552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:40337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/krb5kdc/oryx_127.2.197.195.keytab
--principal=oryx/127.2.197.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.195:42923
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.197.195
--webserver_port=36209
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:37333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.195
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:04:49.957103 12552 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:04:49.957309 12552 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:04:49.959868 12558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:49.959882 12560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:04:49.959885 12557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:04:49.960198 12552 server_base.cc:1061] running on GCE node
I20260429 14:04:49.960413 12552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:04:49.960662 12552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:04:49.961848 12552 hybrid_clock.cc:648] HybridClock initialized: now 1777471489961833 us; error 45 us; skew 500 ppm
Apr 29 14:04:49 dist-test-slave-8n77 krb5kdc[11668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471489, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.197.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:04:49.964520 12552 init.cc:377] Logged in from keytab as oryx/127.2.197.195@KRBTEST.COM (short username oryx)
I20260429 14:04:49.965132 12552 webserver.cc:492] Webserver started at http://127.2.197.195:36209/ using document root <none> and password file <none>
I20260429 14:04:49.965312 12552 fs_manager.cc:362] Metadata directory not provided
I20260429 14:04:49.965363 12552 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:04:49.966656 12552 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:04:49.967214 12567 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:49.967427 12552 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:04:49.967494 12552 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-2/wal
uuid: "83d3d17ea64a49288f23ae9a20f84b87"
format_stamp: "Formatted at 2026-04-29 14:04:49 on dist-test-slave-8n77"
server_key: "cd3ae26c428c5481692b2e94ccf2a120"
server_key_iv: "4cbebfba63a3dcd36ddf4fafd2ab9da8"
server_key_version: "encryptionkey@0"
I20260429 14:04:49.967782 12552 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:04:49.984464 12552 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:04:49.986506 12552 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:04:49.986632 12552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:04:49.986817 12552 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:04:49.987146 12552 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:04:49.987178 12552 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:49.987200 12552 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:04:49.987248 12552 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:04:49.992965 12552 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.195:42923
I20260429 14:04:49.993063 12681 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.195:42923 every 8 connection(s)
I20260429 14:04:49.993353 12552 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
I20260429 14:04:49.995743 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 12552
Apr 29 14:04:49 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.195@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:04:49 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.195@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:04:50.000628 12684 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:49.994104 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.195:57199)
0429 14:04:49.994246 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:49.994308 (+ 62us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:49.994764 (+ 456us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:49.994987 (+ 223us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:49.994990 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:49.995274 (+ 284us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:49.995529 (+ 255us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:49.995532 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:49.996416 (+ 884us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:49.996417 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:49.996972 (+ 555us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:49.996975 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:49.997110 (+ 135us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:49.998022 (+ 912us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:49.998031 (+ 9us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:50.000465 (+ 2434us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":83,"threads_started":1}
W20260429 14:04:50.000659 12418 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:49.994109 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:57199 (local address 127.2.197.254:37333)
0429 14:04:49.994165 (+ 56us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:49.994167 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:49.994395 (+ 228us) server_negotiation.cc:408] Connection header received
0429 14:04:49.994843 (+ 448us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:49.994844 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:49.994853 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:49.994924 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:49.995667 (+ 743us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:49.996324 (+ 657us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:49.997060 (+ 736us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:49.997171 (+ 111us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:50.000615 (+ 3444us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.195:57199: BlockingRecv error: recv got EOF from 127.2.197.195:57199 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260429 14:04:50.001039 12682 heartbeater.cc:646] Failed to heartbeat to 127.2.197.254:37333 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.197.254:37333: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
I20260429 14:04:50.003482 12686 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:49.996418 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33620 (local address 127.2.197.254:37333)
0429 14:04:49.996539 (+ 121us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:49.996541 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:49.996547 (+ 6us) server_negotiation.cc:408] Connection header received
0429 14:04:49.996563 (+ 16us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:49.996564 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:49.996573 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:49.996626 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:49.997541 (+ 915us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:49.998048 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:49.998880 (+ 832us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:49.999045 (+ 165us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:50.000108 (+ 1063us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:04:50.000245 (+ 137us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:04:50.000248 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:04:50.000268 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:04:50.002174 (+ 1906us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:50.002560 (+ 386us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:50.002567 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:50.002567 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:50.002614 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:04:50.002794 (+ 180us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:04:50.002795 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:04:50.002796 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:04:50.003005 (+ 209us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:04:50.003140 (+ 135us) server_negotiation.cc:1036] Waiting for connection context
0429 14:04:50.003234 (+ 94us) server_negotiation.cc:300] Negotiation successful
0429 14:04:50.003358 (+ 124us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":78,"threads_started":1}
Apr 29 14:04:49 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.195@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:04:49 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.195@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:04:50.006052 12684 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:50.001515 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.195:52483)
0429 14:04:50.001590 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:50.001603 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:50.001744 (+ 141us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:50.001938 (+ 194us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:50.001939 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:50.002071 (+ 132us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:50.002237 (+ 166us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:50.002240 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:50.002844 (+ 604us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:50.002845 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:50.003379 (+ 534us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:50.003382 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:50.003465 (+ 83us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:50.004089 (+ 624us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:50.004094 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:50.006003 (+ 1909us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260429 14:04:50.006199 12418 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:50.001685 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:52483 (local address 127.2.197.254:37333)
0429 14:04:50.001735 (+ 50us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:50.001738 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:50.001747 (+ 9us) server_negotiation.cc:408] Connection header received
0429 14:04:50.001854 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:50.001855 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:50.001865 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:50.001935 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:50.002338 (+ 403us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:50.002785 (+ 447us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:50.003511 (+ 726us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:50.003924 (+ 413us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:50.006136 (+ 2212us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.195:52483: BlockingRecv error: recv got EOF from 127.2.197.195:52483 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Apr 29 14:04:50 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.195@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:04:50 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.195@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:04:50.010187 12684 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:50.006386 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.195:52169)
0429 14:04:50.006434 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:50.006444 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:50.006483 (+ 39us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:50.006623 (+ 140us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:50.006624 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:50.006728 (+ 104us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:50.006851 (+ 123us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:50.006852 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:50.007526 (+ 674us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:50.007527 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:50.007904 (+ 377us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:50.007905 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:50.008033 (+ 128us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:50.008472 (+ 439us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:50.008477 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:50.010136 (+ 1659us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260429 14:04:50.010393 12418 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:50.006449 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:52169 (local address 127.2.197.254:37333)
0429 14:04:50.006505 (+ 56us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:50.006507 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:50.006514 (+ 7us) server_negotiation.cc:408] Connection header received
0429 14:04:50.006532 (+ 18us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:50.006533 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:50.006543 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:50.006607 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:50.006952 (+ 345us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:50.007436 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:50.008043 (+ 607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:50.008161 (+ 118us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:50.010258 (+ 2097us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.195:52169: BlockingRecv error: recv got EOF from 127.2.197.195:52169 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260429 14:04:50.010421 12682 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Apr 29 14:04:50 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:04:50 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:04:50.763088 12687 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:50.758515 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.193:34839)
0429 14:04:50.758685 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:50.758699 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:50.758764 (+ 65us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:50.758982 (+ 218us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:50.758983 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:50.759143 (+ 160us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:50.759332 (+ 189us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:50.759334 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:50.760106 (+ 772us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:50.760106 (+ 0us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:50.760467 (+ 361us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:50.760468 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:50.760526 (+ 58us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:50.761049 (+ 523us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:50.761054 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:50.763021 (+ 1967us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":62,"threads_started":1}
W20260429 14:04:50.763281 12688 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:50.758705 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:34839 (local address 127.2.197.254:37333)
0429 14:04:50.758855 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:50.758857 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:50.758867 (+ 10us) server_negotiation.cc:408] Connection header received
0429 14:04:50.758887 (+ 20us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:50.758888 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:50.758898 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:50.758965 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:50.759533 (+ 568us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:50.760032 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:50.760603 (+ 571us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:50.760729 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:50.763177 (+ 2448us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.193:34839: BlockingRecv error: recv got EOF from 127.2.197.193:34839 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":68,"threads_started":1}
Apr 29 14:04:50 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:04:50 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:04:50.891410 12689 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:50.887085 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.194:35553)
0429 14:04:50.887267 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:50.887289 (+ 22us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:50.887362 (+ 73us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:50.887526 (+ 164us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:50.887527 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:50.887687 (+ 160us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:50.887864 (+ 177us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:50.887865 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:50.888495 (+ 630us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:50.888497 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:50.888900 (+ 403us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:50.888902 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:50.888967 (+ 65us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:50.889489 (+ 522us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:50.889494 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:50.891334 (+ 1840us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":134,"threads_started":1}
W20260429 14:04:50.891507 12688 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:50.887183 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:35553 (local address 127.2.197.254:37333)
0429 14:04:50.887265 (+ 82us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:50.887268 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:50.887406 (+ 138us) server_negotiation.cc:408] Connection header received
0429 14:04:50.887434 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:50.887435 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:50.887448 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:50.887493 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:50.887931 (+ 438us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:50.888403 (+ 472us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:50.888979 (+ 576us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:50.889096 (+ 117us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:50.891440 (+ 2344us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.194:35553: BlockingRecv error: recv got EOF from 127.2.197.194:35553 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Apr 29 14:04:51 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.195@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:04:51 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.195@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:04:51.015668 12690 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:51.010904 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.195:40763)
0429 14:04:51.011131 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:51.011143 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:51.011205 (+ 62us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:51.011416 (+ 211us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:51.011417 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:51.011595 (+ 178us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:51.011783 (+ 188us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:51.011785 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:51.012503 (+ 718us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:51.012505 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:51.012915 (+ 410us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:51.012917 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:51.012985 (+ 68us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:51.013630 (+ 645us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:51.013635 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:51.015600 (+ 1965us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":132,"threads_started":1}
W20260429 14:04:51.015808 12688 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:51.011065 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:40763 (local address 127.2.197.254:37333)
0429 14:04:51.011142 (+ 77us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:51.011145 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:51.011261 (+ 116us) server_negotiation.cc:408] Connection header received
0429 14:04:51.011292 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:51.011293 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:51.011308 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:51.011369 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:51.011914 (+ 545us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:51.012396 (+ 482us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:51.013018 (+ 622us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:51.013142 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:51.015714 (+ 2572us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.195:40763: BlockingRecv error: recv got EOF from 127.2.197.195:40763 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Apr 29 14:04:51 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:04:51 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:04:51.771610 12691 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:51.763619 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.193:33083)
0429 14:04:51.763771 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:51.763781 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:51.763831 (+ 50us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:51.764092 (+ 261us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:51.764094 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:51.764258 (+ 164us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:51.764413 (+ 155us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:51.764414 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:51.765138 (+ 724us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:51.765139 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:51.765513 (+ 374us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:51.765514 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:51.765582 (+ 68us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:51.769729 (+ 4147us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:51.769734 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:51.771547 (+ 1813us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":87,"threads_started":1}
W20260429 14:04:51.771840 12692 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:51.763733 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:33083 (local address 127.2.197.254:37333)
0429 14:04:51.763944 (+ 211us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:51.763946 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:51.763954 (+ 8us) server_negotiation.cc:408] Connection header received
0429 14:04:51.763975 (+ 21us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:51.763975 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:51.763986 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:51.764048 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:51.764507 (+ 459us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:51.765060 (+ 553us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:51.765615 (+ 555us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:51.765740 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:51.771759 (+ 6019us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.193:33083: BlockingRecv error: recv got EOF from 127.2.197.193:33083 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":157,"threads_started":1}
Apr 29 14:04:51 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:04:51 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:04:51.898393 12693 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:51.892057 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.194:35353)
0429 14:04:51.892230 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:51.892243 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:51.892305 (+ 62us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:51.892487 (+ 182us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:51.892489 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:51.892701 (+ 212us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:51.892953 (+ 252us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:51.892956 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:51.894216 (+ 1260us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:51.894217 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:51.895061 (+ 844us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:51.895066 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:51.895175 (+ 109us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:51.895731 (+ 556us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:51.895737 (+ 6us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:51.898314 (+ 2577us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":90,"threads_started":1}
W20260429 14:04:51.898507 12692 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:51.892209 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:35353 (local address 127.2.197.254:37333)
0429 14:04:51.892267 (+ 58us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:51.892270 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:51.892291 (+ 21us) server_negotiation.cc:408] Connection header received
0429 14:04:51.892375 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:51.892376 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:51.892391 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:51.892476 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:51.893104 (+ 628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:51.894067 (+ 963us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:51.895231 (+ 1164us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:51.895497 (+ 266us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:51.898447 (+ 2950us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.194:35353: BlockingRecv error: recv got EOF from 127.2.197.194:35353 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Apr 29 14:04:52 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.195@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:04:52 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.195@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:04:52.021080 12694 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:52.016252 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.195:48345)
0429 14:04:52.016421 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:52.016435 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:52.016520 (+ 85us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:52.016723 (+ 203us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:52.016724 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:52.016929 (+ 205us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:52.017087 (+ 158us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:52.017089 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:52.017761 (+ 672us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:52.017762 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:52.018137 (+ 375us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:52.018138 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:52.018224 (+ 86us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:52.018916 (+ 692us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:52.018922 (+ 6us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:52.020984 (+ 2062us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":64,"threads_started":1}
W20260429 14:04:52.021131 12692 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:52.016476 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:48345 (local address 127.2.197.254:37333)
0429 14:04:52.016598 (+ 122us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:52.016601 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:52.016609 (+ 8us) server_negotiation.cc:408] Connection header received
0429 14:04:52.016641 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:52.016642 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:52.016651 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:52.016715 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:52.017181 (+ 466us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:52.017673 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:52.018223 (+ 550us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:52.018429 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:52.021072 (+ 2643us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.195:48345: BlockingRecv error: recv got EOF from 127.2.197.195:48345 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Apr 29 14:04:52 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:04:52 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:04:52.776587 12695 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:52.772261 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.193:33973)
0429 14:04:52.772446 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:52.772456 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:52.772540 (+ 84us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:52.772708 (+ 168us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:52.772713 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:52.772961 (+ 248us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:52.773116 (+ 155us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:52.773118 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:52.773841 (+ 723us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:52.773842 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:52.774220 (+ 378us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:52.774222 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:52.774350 (+ 128us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:52.774863 (+ 513us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:52.774868 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:52.776526 (+ 1658us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":111,"threads_started":1}
W20260429 14:04:52.776777 12696 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:52.772350 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:33973 (local address 127.2.197.254:37333)
0429 14:04:52.772545 (+ 195us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:52.772547 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:52.772558 (+ 11us) server_negotiation.cc:408] Connection header received
0429 14:04:52.772588 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:52.772589 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:52.772604 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:52.772702 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:52.773194 (+ 492us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:52.773757 (+ 563us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:52.774349 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:52.774475 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:52.776697 (+ 2222us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.193:33973: BlockingRecv error: recv got EOF from 127.2.197.193:33973 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
Apr 29 14:04:52 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:04:52 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:04:52.903537 12697 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:52.899082 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.194:38653)
0429 14:04:52.899273 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:52.899284 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:52.899350 (+ 66us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:52.899690 (+ 340us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:52.899692 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:52.899855 (+ 163us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:52.900013 (+ 158us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:52.900014 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:52.900680 (+ 666us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:52.900681 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:52.901065 (+ 384us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:52.901066 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:52.901127 (+ 61us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:52.901636 (+ 509us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:52.901640 (+ 4us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:52.903476 (+ 1836us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":86,"threads_started":1}
W20260429 14:04:52.903676 12696 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:52.899209 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:38653 (local address 127.2.197.254:37333)
0429 14:04:52.899302 (+ 93us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:52.899304 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:52.899333 (+ 29us) server_negotiation.cc:408] Connection header received
0429 14:04:52.899425 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:52.899426 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:52.899437 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:52.899634 (+ 197us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:52.900105 (+ 471us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:52.900601 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:52.901132 (+ 531us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:52.901241 (+ 109us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:52.903601 (+ 2360us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.194:38653: BlockingRecv error: recv got EOF from 127.2.197.194:38653 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Apr 29 14:04:53 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.195@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:04:53 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.195@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:04:53.027295 12698 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:53.021715 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.195:38043)
0429 14:04:53.021882 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:53.021894 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:53.021971 (+ 77us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:53.022226 (+ 255us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:53.022228 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:53.022537 (+ 309us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:53.022743 (+ 206us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:53.022746 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:53.023615 (+ 869us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:53.023617 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:53.024032 (+ 415us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:53.024034 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:53.024108 (+ 74us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:53.024700 (+ 592us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:53.024705 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:53.027221 (+ 2516us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":124,"threads_started":1}
W20260429 14:04:53.027895 12696 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:53.021773 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:38043 (local address 127.2.197.254:37333)
0429 14:04:53.021844 (+ 71us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:53.021847 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:53.021968 (+ 121us) server_negotiation.cc:408] Connection header received
0429 14:04:53.022028 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:53.022029 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:53.022039 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:53.022097 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:53.022929 (+ 832us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:53.023464 (+ 535us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:53.024136 (+ 672us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:53.024261 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:53.027758 (+ 3497us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.195:38043: BlockingRecv error: recv got EOF from 127.2.197.195:38043 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Apr 29 14:04:53 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:04:53 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:04:53.781816 12699 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:53.777179 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.193:36627)
0429 14:04:53.777365 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:53.777377 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:53.777448 (+ 71us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:53.777632 (+ 184us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:53.777633 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:53.777804 (+ 171us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:53.777962 (+ 158us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:53.777965 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:53.778753 (+ 788us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:53.778755 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:53.779149 (+ 394us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:53.779151 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:53.779218 (+ 67us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:53.779751 (+ 533us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:53.779756 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:53.781751 (+ 1995us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":121,"threads_started":1}
W20260429 14:04:53.781993 12700 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:53.777266 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:36627 (local address 127.2.197.254:37333)
0429 14:04:53.777377 (+ 111us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:53.777379 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:53.777447 (+ 68us) server_negotiation.cc:408] Connection header received
0429 14:04:53.777539 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:53.777540 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:53.777551 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:53.777613 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:53.778057 (+ 444us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:53.778635 (+ 578us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:53.779228 (+ 593us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:53.779349 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:53.781907 (+ 2558us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.193:36627: BlockingRecv error: recv got EOF from 127.2.197.193:36627 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":48,"threads_started":1}
Apr 29 14:04:53 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:04:53 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:04:53.909121 12701 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:53.904283 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.194:36645)
0429 14:04:53.904416 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:53.904435 (+ 19us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:53.904483 (+ 48us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:53.904698 (+ 215us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:53.904699 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:53.904877 (+ 178us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:53.905048 (+ 171us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:53.905049 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:53.905864 (+ 815us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:53.905866 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:53.906577 (+ 711us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:53.906581 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:53.906671 (+ 90us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:53.907191 (+ 520us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:53.907196 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:53.909025 (+ 1829us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
W20260429 14:04:53.909169 12700 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:53.904424 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:36645 (local address 127.2.197.254:37333)
0429 14:04:53.904485 (+ 61us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:53.904487 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:53.904500 (+ 13us) server_negotiation.cc:408] Connection header received
0429 14:04:53.904546 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:53.904547 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:53.904583 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:53.904640 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:53.905158 (+ 518us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:53.905700 (+ 542us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:53.906685 (+ 985us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:53.906894 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:53.909108 (+ 2214us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.194:36645: BlockingRecv error: recv got EOF from 127.2.197.194:36645 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Apr 29 14:04:54 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.195@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:04:54 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.195@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:04:54.032222 12702 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:54.027818 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.195:33457)
0429 14:04:54.027959 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:54.027970 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:54.028028 (+ 58us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:54.028154 (+ 126us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:54.028155 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:54.028314 (+ 159us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:54.028471 (+ 157us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:54.028473 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:54.029158 (+ 685us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:54.029159 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:54.029566 (+ 407us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:54.029568 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:54.029644 (+ 76us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:54.030225 (+ 581us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:54.030229 (+ 4us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:54.032163 (+ 1934us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":94,"threads_started":1}
W20260429 14:04:54.032367 12700 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:54.027876 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:33457 (local address 127.2.197.254:37333)
0429 14:04:54.027920 (+ 44us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:54.027923 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:54.028049 (+ 126us) server_negotiation.cc:408] Connection header received
0429 14:04:54.028075 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:54.028076 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:54.028085 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:54.028140 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:54.028589 (+ 449us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:54.029063 (+ 474us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:54.029688 (+ 625us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:54.029825 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:54.032289 (+ 2464us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.195:33457: BlockingRecv error: recv got EOF from 127.2.197.195:33457 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Apr 29 14:04:54 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:04:54 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:04:54.787408 12703 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:54.782386 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.193:58527)
0429 14:04:54.782510 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:54.782529 (+ 19us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:54.782590 (+ 61us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:54.782741 (+ 151us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:54.782742 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:54.782950 (+ 208us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:54.783104 (+ 154us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:54.783106 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:54.784143 (+ 1037us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:54.784145 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:54.784803 (+ 658us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:54.784809 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:54.784889 (+ 80us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:54.785381 (+ 492us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:54.785386 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:54.787347 (+ 1961us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
W20260429 14:04:54.787590 12704 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:54.782441 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:58527 (local address 127.2.197.254:37333)
0429 14:04:54.782579 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:54.782581 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:54.782591 (+ 10us) server_negotiation.cc:408] Connection header received
0429 14:04:54.782635 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:54.782636 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:54.782648 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:54.782730 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:54.783191 (+ 461us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:54.783834 (+ 643us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:54.784913 (+ 1079us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:54.785041 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:54.787518 (+ 2477us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.193:58527: BlockingRecv error: recv got EOF from 127.2.197.193:58527 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":85,"threads_started":1}
Apr 29 14:04:54 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:04:54 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:04:54.913942 12705 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:54.909645 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.194:35615)
0429 14:04:54.909790 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:54.909801 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:54.909894 (+ 93us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:54.910054 (+ 160us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:54.910055 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:54.910237 (+ 182us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:54.910410 (+ 173us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:54.910412 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:54.911091 (+ 679us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:54.911092 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:54.911474 (+ 382us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:54.911476 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:54.911562 (+ 86us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:54.912081 (+ 519us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:54.912085 (+ 4us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:54.913843 (+ 1758us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
W20260429 14:04:54.913993 12704 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:54.909820 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:35615 (local address 127.2.197.254:37333)
0429 14:04:54.909887 (+ 67us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:54.909889 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:54.909900 (+ 11us) server_negotiation.cc:408] Connection header received
0429 14:04:54.909955 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:54.909956 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:54.909969 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:54.910046 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:54.910501 (+ 455us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:54.911009 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:54.911557 (+ 548us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:54.911752 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:54.913932 (+ 2180us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.194:35615: BlockingRecv error: recv got EOF from 127.2.197.194:35615 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Apr 29 14:04:55 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.195@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:04:55 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.195@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:04:55.037801 12706 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:55.032839 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.195:60897)
0429 14:04:55.032990 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:55.033004 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:55.033061 (+ 57us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:55.033323 (+ 262us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:55.033324 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:55.033541 (+ 217us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:55.033697 (+ 156us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:55.033699 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:55.034486 (+ 787us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:55.034488 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:55.034874 (+ 386us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:55.034875 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:55.034953 (+ 78us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:55.035518 (+ 565us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:55.035522 (+ 4us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:55.037712 (+ 2190us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":97,"threads_started":1}
W20260429 14:04:55.037962 12704 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:55.032987 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:60897 (local address 127.2.197.254:37333)
0429 14:04:55.033049 (+ 62us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:55.033052 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:55.033065 (+ 13us) server_negotiation.cc:408] Connection header received
0429 14:04:55.033192 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:55.033194 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:55.033207 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:55.033279 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:55.033825 (+ 546us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:55.034406 (+ 581us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:55.034971 (+ 565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:55.035094 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:55.037899 (+ 2805us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.195:60897: BlockingRecv error: recv got EOF from 127.2.197.195:60897 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Apr 29 14:04:55 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:04:55 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:04:55.792295 12707 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:55.788056 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.193:35375)
0429 14:04:55.788229 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:55.788242 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:55.788323 (+ 81us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:55.788493 (+ 170us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:55.788494 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:55.788698 (+ 204us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:55.788863 (+ 165us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:55.788865 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:55.789596 (+ 731us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:55.789597 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:55.789986 (+ 389us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:55.789987 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:55.790047 (+ 60us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:55.790614 (+ 567us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:55.790619 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:55.792237 (+ 1618us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":85,"threads_started":1}
W20260429 14:04:55.792454 12708 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:55.788126 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:35375 (local address 127.2.197.254:37333)
0429 14:04:55.788275 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:55.788276 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:55.788305 (+ 29us) server_negotiation.cc:408] Connection header received
0429 14:04:55.788376 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:55.788377 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:55.788388 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:55.788448 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:55.788959 (+ 511us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:55.789507 (+ 548us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:55.790086 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:55.790221 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:55.792378 (+ 2157us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.193:35375: BlockingRecv error: recv got EOF from 127.2.197.193:35375 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":91,"threads_started":1}
Apr 29 14:04:55 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:04:55 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:04:55.919193 12709 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:55.914529 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.194:49421)
0429 14:04:55.914733 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:55.914748 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:55.914843 (+ 95us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:55.914988 (+ 145us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:55.914989 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:55.915181 (+ 192us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:55.915335 (+ 154us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:55.915337 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:55.916014 (+ 677us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:55.916015 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:55.916390 (+ 375us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:55.916391 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:55.916472 (+ 81us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:55.917006 (+ 534us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:55.917011 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:55.919096 (+ 2085us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":113,"threads_started":1}
W20260429 14:04:55.919242 12708 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:55.914653 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:49421 (local address 127.2.197.254:37333)
0429 14:04:55.914738 (+ 85us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:55.914740 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:55.914828 (+ 88us) server_negotiation.cc:408] Connection header received
0429 14:04:55.914906 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:55.914907 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:55.914922 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:55.914983 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:55.915420 (+ 437us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:55.915922 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:55.916468 (+ 546us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:55.916653 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:55.919182 (+ 2529us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.194:49421: BlockingRecv error: recv got EOF from 127.2.197.194:49421 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Apr 29 14:04:56 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.195@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:04:56 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.195@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:04:56.043388 12710 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:56.038578 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.195:56465)
0429 14:04:56.038786 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:56.038800 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:56.038903 (+ 103us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:56.039094 (+ 191us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:56.039095 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:56.039296 (+ 201us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:56.039463 (+ 167us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:56.039465 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:56.040268 (+ 803us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:56.040269 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:56.040774 (+ 505us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:56.040776 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:56.040871 (+ 95us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:56.041350 (+ 479us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:56.041355 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:56.043293 (+ 1938us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":106,"threads_started":1}
W20260429 14:04:56.043454 12708 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:56.038677 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:56465 (local address 127.2.197.254:37333)
0429 14:04:56.038784 (+ 107us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:56.038786 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:56.038880 (+ 94us) server_negotiation.cc:408] Connection header received
0429 14:04:56.038987 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:56.038988 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:56.039002 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:56.039079 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:56.039556 (+ 477us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:56.040166 (+ 610us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:56.040870 (+ 704us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:56.041056 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:56.043377 (+ 2321us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.195:56465: BlockingRecv error: recv got EOF from 127.2.197.195:56465 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Apr 29 14:04:56 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:04:56 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:04:56.802800 12711 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:56.792905 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.193:46775)
0429 14:04:56.793058 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:56.793070 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:56.793197 (+ 127us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:56.793493 (+ 296us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:56.793495 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:56.793701 (+ 206us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:56.793946 (+ 245us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:56.793948 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:56.794870 (+ 922us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:56.794871 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:56.795261 (+ 390us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:56.795263 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:56.795328 (+ 65us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:56.800982 (+ 5654us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:56.800987 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:56.802741 (+ 1754us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":92,"threads_started":1}
W20260429 14:04:56.803000 12712 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:56.793055 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:46775 (local address 127.2.197.254:37333)
0429 14:04:56.793242 (+ 187us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:56.793244 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:56.793255 (+ 11us) server_negotiation.cc:408] Connection header received
0429 14:04:56.793340 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:56.793342 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:56.793356 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:56.793440 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:56.794156 (+ 716us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:56.794793 (+ 637us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:56.795335 (+ 542us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:56.795449 (+ 114us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:56.802936 (+ 7487us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.193:46775: BlockingRecv error: recv got EOF from 127.2.197.193:46775 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":78,"threads_started":1}
Apr 29 14:04:56 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:04:56 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:04:56.924073 12713 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:56.919757 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.194:41565)
0429 14:04:56.919888 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:56.919898 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:56.919956 (+ 58us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:56.920076 (+ 120us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:56.920077 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:56.920239 (+ 162us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:56.920396 (+ 157us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:56.920398 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:56.921128 (+ 730us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:56.921129 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:56.921545 (+ 416us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:56.921547 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:56.921619 (+ 72us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:56.922178 (+ 559us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:56.922183 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:56.924015 (+ 1832us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":81,"threads_started":1}
W20260429 14:04:56.924187 12712 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:56.919817 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:41565 (local address 127.2.197.254:37333)
0429 14:04:56.919895 (+ 78us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:56.919897 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:56.919953 (+ 56us) server_negotiation.cc:408] Connection header received
0429 14:04:56.919999 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:56.919999 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:56.920009 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:56.920064 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:56.920497 (+ 433us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:56.921032 (+ 535us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:56.921627 (+ 595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:56.921755 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:56.924113 (+ 2358us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.194:41565: BlockingRecv error: recv got EOF from 127.2.197.194:41565 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Apr 29 14:04:57 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.195@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:04:57 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.195@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:04:57.048138 12714 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:57.043986 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.195:55047)
0429 14:04:57.044134 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:57.044145 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:57.044204 (+ 59us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:57.044341 (+ 137us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:57.044342 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:57.044539 (+ 197us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:57.044701 (+ 162us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:57.044702 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:57.045348 (+ 646us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:57.045349 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:57.045759 (+ 410us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:57.045760 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:57.045817 (+ 57us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:57.046335 (+ 518us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:57.046339 (+ 4us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:57.048079 (+ 1740us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":92,"threads_started":1}
W20260429 14:04:57.048323 12712 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:57.044080 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:55047 (local address 127.2.197.254:37333)
0429 14:04:57.044139 (+ 59us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:57.044141 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:57.044203 (+ 62us) server_negotiation.cc:408] Connection header received
0429 14:04:57.044247 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:57.044248 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:57.044257 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:57.044314 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:57.044796 (+ 482us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:57.045271 (+ 475us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:57.045837 (+ 566us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:57.045953 (+ 116us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:57.048206 (+ 2253us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.195:55047: BlockingRecv error: recv got EOF from 127.2.197.195:55047 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Apr 29 14:04:57 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:04:57 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:04:57.808463 12715 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:57.803423 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.193:46517)
0429 14:04:57.803623 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:57.803636 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:57.803706 (+ 70us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:57.803911 (+ 205us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:57.803912 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:57.804076 (+ 164us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:57.804237 (+ 161us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:57.804239 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:57.805065 (+ 826us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:57.805066 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:57.805471 (+ 405us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:57.805472 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:57.805540 (+ 68us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:57.806128 (+ 588us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:57.806133 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:57.808403 (+ 2270us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
W20260429 14:04:57.808573 12716 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:57.803608 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:46517 (local address 127.2.197.254:37333)
0429 14:04:57.803752 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:57.803754 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:57.803764 (+ 10us) server_negotiation.cc:408] Connection header received
0429 14:04:57.803808 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:57.803808 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:57.803820 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:57.803887 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:57.804329 (+ 442us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:57.804966 (+ 637us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:57.805617 (+ 651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:57.805756 (+ 139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:57.808521 (+ 2765us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.193:46517: BlockingRecv error: recv got EOF from 127.2.197.193:46517 (error 108)
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":69,"threads_started":1}
Apr 29 14:04:57 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:04:57 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:04:57.928778 12717 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:57.924567 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.194:60595)
0429 14:04:57.924712 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:57.924722 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:57.924778 (+ 56us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:57.924899 (+ 121us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:57.924900 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:57.925066 (+ 166us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:57.925225 (+ 159us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:57.925227 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:57.926090 (+ 863us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:57.926092 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:57.926499 (+ 407us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:57.926500 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:57.926563 (+ 63us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:57.927083 (+ 520us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:57.927088 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:57.928720 (+ 1632us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
W20260429 14:04:57.928867 12716 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:57.924664 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:60595 (local address 127.2.197.254:37333)
0429 14:04:57.924721 (+ 57us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:57.924723 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:57.924770 (+ 47us) server_negotiation.cc:408] Connection header received
0429 14:04:57.924820 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:57.924820 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:57.924830 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:57.924884 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:57.925310 (+ 426us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:57.926006 (+ 696us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:57.926597 (+ 591us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:57.926722 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:57.928824 (+ 2102us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.194:60595: BlockingRecv error: recv got EOF from 127.2.197.194:60595 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Apr 29 14:04:58 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.195@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:04:58 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.195@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:04:58.053191 12718 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:58.048703 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.195:59373)
0429 14:04:58.049070 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:58.049080 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:58.049288 (+ 208us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:58.049464 (+ 176us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:58.049465 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:58.049651 (+ 186us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:58.049807 (+ 156us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:58.049809 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:58.050523 (+ 714us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:58.050524 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:58.050903 (+ 379us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:58.050905 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:58.050991 (+ 86us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:58.051490 (+ 499us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:58.051495 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:58.053134 (+ 1639us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":299,"threads_started":1}
W20260429 14:04:58.053365 12716 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:58.048859 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:59373 (local address 127.2.197.254:37333)
0429 14:04:58.048930 (+ 71us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:58.048932 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:58.049267 (+ 335us) server_negotiation.cc:408] Connection header received
0429 14:04:58.049368 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:58.049369 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:58.049380 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:58.049443 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:58.049915 (+ 472us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:58.050448 (+ 533us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:58.050987 (+ 539us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:58.051173 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:58.053256 (+ 2083us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.195:59373: BlockingRecv error: recv got EOF from 127.2.197.195:59373 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Apr 29 14:04:58 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:04:58 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:04:58.813485 12719 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:58.809129 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.193:49659)
0429 14:04:58.809262 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:58.809275 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:58.809343 (+ 68us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:58.809532 (+ 189us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:58.809533 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:58.809722 (+ 189us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:58.809927 (+ 205us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:58.809929 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:58.810617 (+ 688us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:58.810618 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:58.810997 (+ 379us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:58.810999 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:58.811057 (+ 58us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:58.811585 (+ 528us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:58.811590 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:58.813425 (+ 1835us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
W20260429 14:04:58.813684 12720 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:58.809208 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:49659 (local address 127.2.197.254:37333)
0429 14:04:58.809322 (+ 114us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:58.809324 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:58.809335 (+ 11us) server_negotiation.cc:408] Connection header received
0429 14:04:58.809424 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:58.809425 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:58.809438 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:58.809509 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:58.810000 (+ 491us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:58.810544 (+ 544us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:58.811148 (+ 604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:58.811346 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:58.813619 (+ 2273us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.193:49659: BlockingRecv error: recv got EOF from 127.2.197.193:49659 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":68,"threads_started":1}
Apr 29 14:04:58 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:04:58 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:04:58.934058 12721 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:58.929335 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.194:56183)
0429 14:04:58.929634 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:58.929646 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:58.929746 (+ 100us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:58.929919 (+ 173us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:58.929920 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:58.930103 (+ 183us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:58.930255 (+ 152us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:58.930256 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:58.931097 (+ 841us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:58.931098 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:58.931464 (+ 366us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:58.931466 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:58.931562 (+ 96us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:58.932088 (+ 526us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:58.932092 (+ 4us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:58.933999 (+ 1907us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":104,"threads_started":1}
W20260429 14:04:58.934229 12720 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:58.929434 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:56183 (local address 127.2.197.254:37333)
0429 14:04:58.929629 (+ 195us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:58.929632 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:58.929723 (+ 91us) server_negotiation.cc:408] Connection header received
0429 14:04:58.929832 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:58.929833 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:58.929844 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:58.929914 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:58.930418 (+ 504us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:58.930996 (+ 578us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:58.931582 (+ 586us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:58.931710 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:58.934153 (+ 2443us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.194:56183: BlockingRecv error: recv got EOF from 127.2.197.194:56183 (error 108)
Metrics: {"server-negotiator.queue_time_us":162}
Apr 29 14:04:59 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.195@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:04:59 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.195@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:04:59.058223 12722 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:59.053703 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.195:52529)
0429 14:04:59.053859 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:59.053870 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:59.053919 (+ 49us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:59.054126 (+ 207us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:59.054128 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:59.054338 (+ 210us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:59.054520 (+ 182us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:59.054522 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:59.055185 (+ 663us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:59.055186 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:59.055576 (+ 390us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:59.055577 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:59.055643 (+ 66us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:59.056184 (+ 541us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:59.056188 (+ 4us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:59.058167 (+ 1979us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":101,"threads_started":1}
W20260429 14:04:59.058389 12720 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:59.053829 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:52529 (local address 127.2.197.254:37333)
0429 14:04:59.053892 (+ 63us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:59.053894 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:59.053907 (+ 13us) server_negotiation.cc:408] Connection header received
0429 14:04:59.054013 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:59.054014 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:59.054025 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:59.054094 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:59.054608 (+ 514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:59.055099 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:59.055667 (+ 568us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:59.055802 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:59.058269 (+ 2467us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.195:52529: BlockingRecv error: recv got EOF from 127.2.197.195:52529 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Apr 29 14:04:59 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:04:59 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:04:59.818663 12723 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:59.813942 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.193:44541)
0429 14:04:59.814104 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:59.814116 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:59.814172 (+ 56us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:59.814376 (+ 204us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:59.814377 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:59.814542 (+ 165us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:59.814704 (+ 162us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:59.814706 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:59.815381 (+ 675us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:59.815382 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:59.815801 (+ 419us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:59.815802 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:59.815866 (+ 64us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:59.816378 (+ 512us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:59.816383 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:59.818603 (+ 2220us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":98,"threads_started":1}
W20260429 14:04:59.818840 12724 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:59.814081 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:44541 (local address 127.2.197.254:37333)
0429 14:04:59.814204 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:59.814206 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:59.814214 (+ 8us) server_negotiation.cc:408] Connection header received
0429 14:04:59.814233 (+ 19us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:59.814234 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:59.814243 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:59.814366 (+ 123us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:59.814799 (+ 433us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:59.815307 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:59.815966 (+ 659us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:59.816167 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:59.818765 (+ 2598us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.193:44541: BlockingRecv error: recv got EOF from 127.2.197.193:44541 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":78,"threads_started":1}
Apr 29 14:04:59 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:04:59 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:04:59.938982 12725 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:04:59.934788 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.194:45165)
0429 14:04:59.934963 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0429 14:04:59.934975 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:04:59.935051 (+ 76us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:04:59.935215 (+ 164us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:04:59.935215 (+ 0us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:04:59.935401 (+ 186us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:04:59.935552 (+ 151us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:59.935554 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:59.936224 (+ 670us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:04:59.936225 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:04:59.936604 (+ 379us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:04:59.936605 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:04:59.936687 (+ 82us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:59.937180 (+ 493us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:04:59.937184 (+ 4us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:04:59.938887 (+ 1703us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":111,"threads_started":1}
W20260429 14:04:59.939030 12724 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:04:59.934872 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:45165 (local address 127.2.197.254:37333)
0429 14:04:59.934940 (+ 68us) server_negotiation.cc:207] Beginning negotiation
0429 14:04:59.934942 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:04:59.935069 (+ 127us) server_negotiation.cc:408] Connection header received
0429 14:04:59.935121 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:04:59.935122 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:04:59.935135 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:04:59.935208 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:04:59.935643 (+ 435us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:59.936138 (+ 495us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:04:59.936682 (+ 544us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:04:59.936885 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:04:59.938970 (+ 2085us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.194:45165: BlockingRecv error: recv got EOF from 127.2.197.194:45165 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Apr 29 14:05:00 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.195@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:05:00 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.195@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:05:00.063746 12726 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:00.058817 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.195:54707)
0429 14:05:00.059063 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0429 14:05:00.059076 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 14:05:00.059133 (+ 57us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:05:00.059407 (+ 274us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:05:00.059409 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:05:00.059717 (+ 308us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:05:00.059888 (+ 171us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:00.059890 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:00.060552 (+ 662us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:05:00.060554 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:05:00.060931 (+ 377us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:00.060932 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:00.061014 (+ 82us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:00.061571 (+ 557us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:05:00.061580 (+ 9us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:05:00.063658 (+ 2078us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":57,"threads_started":1}
W20260429 14:05:00.063824 12724 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:05:00.059117 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:54707 (local address 127.2.197.254:37333)
0429 14:05:00.059178 (+ 61us) server_negotiation.cc:207] Beginning negotiation
0429 14:05:00.059181 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:05:00.059189 (+ 8us) server_negotiation.cc:408] Connection header received
0429 14:05:00.059215 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:05:00.059216 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:05:00.059229 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:05:00.059401 (+ 172us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:05:00.059981 (+ 580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:00.060474 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:05:00.061011 (+ 537us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:00.061193 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:00.063765 (+ 2572us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.195:54707: BlockingRecv error: recv got EOF from 127.2.197.195:54707 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Apr 29 14:05:00 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:05:00 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:05:00.823623 12727 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:00.819269 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.193:47887)
0429 14:05:00.819422 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0429 14:05:00.819433 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 14:05:00.819490 (+ 57us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:05:00.819653 (+ 163us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:05:00.819653 (+ 0us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:05:00.819830 (+ 177us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:05:00.819983 (+ 153us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:00.819985 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:00.820807 (+ 822us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:05:00.820808 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:05:00.821357 (+ 549us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:00.821359 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:00.821422 (+ 63us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:00.821919 (+ 497us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:05:00.821923 (+ 4us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:05:00.823563 (+ 1640us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":92,"threads_started":1}
W20260429 14:05:00.823843 12728 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:05:00.819380 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:47887 (local address 127.2.197.254:37333)
0429 14:05:00.819525 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0429 14:05:00.819527 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:05:00.819541 (+ 14us) server_negotiation.cc:408] Connection header received
0429 14:05:00.819562 (+ 21us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:05:00.819563 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:05:00.819573 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:05:00.819633 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:05:00.820082 (+ 449us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:00.820697 (+ 615us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:05:00.821428 (+ 731us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:00.821557 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:00.823767 (+ 2210us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.193:47887: BlockingRecv error: recv got EOF from 127.2.197.193:47887 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
Apr 29 14:05:00 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:05:00 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:05:00.943818 12729 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:00.939599 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.194:36457)
0429 14:05:00.939781 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0429 14:05:00.939792 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 14:05:00.939861 (+ 69us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:05:00.940032 (+ 171us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:05:00.940034 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:05:00.940212 (+ 178us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:05:00.940368 (+ 156us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:00.940370 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:00.941081 (+ 711us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:05:00.941083 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:05:00.941462 (+ 379us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:00.941463 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:00.941525 (+ 62us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:00.942057 (+ 532us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:05:00.942062 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:05:00.943751 (+ 1689us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":81,"threads_started":1}
W20260429 14:05:00.943964 12728 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:05:00.939699 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:36457 (local address 127.2.197.254:37333)
0429 14:05:00.939778 (+ 79us) server_negotiation.cc:207] Beginning negotiation
0429 14:05:00.939780 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:05:00.939867 (+ 87us) server_negotiation.cc:408] Connection header received
0429 14:05:00.939916 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:05:00.939917 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:05:00.939927 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:05:00.939978 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:05:00.940476 (+ 498us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:00.940996 (+ 520us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:05:00.941601 (+ 605us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:00.941731 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:00.943875 (+ 2144us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.194:36457: BlockingRecv error: recv got EOF from 127.2.197.194:36457 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Apr 29 14:05:01 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.195@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:05:01 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.195@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
W20260429 14:05:01.069365 12728 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:05:01.064458 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:58805 (local address 127.2.197.254:37333)
0429 14:05:01.064555 (+ 97us) server_negotiation.cc:207] Beginning negotiation
0429 14:05:01.064557 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:05:01.064572 (+ 15us) server_negotiation.cc:408] Connection header received
0429 14:05:01.064665 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:05:01.064666 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:05:01.064675 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:05:01.064733 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:05:01.065170 (+ 437us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:01.065663 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:05:01.066263 (+ 600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:01.066516 (+ 253us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:01.069317 (+ 2801us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.195:58805: BlockingRecv error: recv got EOF from 127.2.197.195:58805 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260429 14:05:01.069366 12730 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:01.064360 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.195:58805)
0429 14:05:01.064520 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0429 14:05:01.064531 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 14:05:01.064590 (+ 59us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:05:01.064748 (+ 158us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:05:01.064749 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:05:01.064930 (+ 181us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:05:01.065086 (+ 156us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:01.065087 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:01.065781 (+ 694us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:05:01.065782 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:05:01.066171 (+ 389us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:01.066172 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:01.066239 (+ 67us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:01.066788 (+ 549us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:05:01.066792 (+ 4us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:05:01.069203 (+ 2411us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":109,"threads_started":1}
Apr 29 14:05:01 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:05:01 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:05:01.831610 12731 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:01.824159 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.193:38347)
0429 14:05:01.824320 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0429 14:05:01.824337 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0429 14:05:01.824397 (+ 60us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:05:01.824529 (+ 132us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:05:01.824529 (+ 0us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:05:01.824703 (+ 174us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:05:01.824870 (+ 167us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:01.824872 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:01.825600 (+ 728us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:05:01.825601 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:05:01.825986 (+ 385us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:01.825988 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:01.826050 (+ 62us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:01.829697 (+ 3647us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:05:01.829702 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:05:01.831550 (+ 1848us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":78,"threads_started":1}
W20260429 14:05:01.831825 12732 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:05:01.824217 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:38347 (local address 127.2.197.254:37333)
0429 14:05:01.824380 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0429 14:05:01.824382 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:05:01.824394 (+ 12us) server_negotiation.cc:408] Connection header received
0429 14:05:01.824443 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:05:01.824444 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:05:01.824454 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:05:01.824515 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:05:01.824988 (+ 473us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:01.825508 (+ 520us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:05:01.826062 (+ 554us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:01.826175 (+ 113us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:01.831768 (+ 5593us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.193:38347: BlockingRecv error: recv got EOF from 127.2.197.193:38347 (error 108)
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":86,"threads_started":1}
Apr 29 14:05:01 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:05:01 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:05:01.950953 12734 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:01.944549 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.194:52095)
0429 14:05:01.944831 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0429 14:05:01.944843 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:05:01.944907 (+ 64us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:05:01.945205 (+ 298us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:05:01.945207 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:05:01.945387 (+ 180us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:05:01.945577 (+ 190us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:01.945579 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:01.946621 (+ 1042us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:05:01.946622 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:05:01.947073 (+ 451us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:01.947075 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:01.947147 (+ 72us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:01.947847 (+ 700us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:05:01.947852 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:05:01.950872 (+ 3020us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":78,"threads_started":1}
W20260429 14:05:01.951056 12732 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:05:01.944768 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:52095 (local address 127.2.197.254:37333)
0429 14:05:01.944941 (+ 173us) server_negotiation.cc:207] Beginning negotiation
0429 14:05:01.944944 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:05:01.944962 (+ 18us) server_negotiation.cc:408] Connection header received
0429 14:05:01.945009 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:05:01.945011 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:05:01.945029 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:05:01.945125 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:05:01.945723 (+ 598us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:01.946313 (+ 590us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:05:01.947194 (+ 881us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:01.947374 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:01.950992 (+ 3618us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.194:52095: BlockingRecv error: recv got EOF from 127.2.197.194:52095 (error 108)
Metrics: {"server-negotiator.queue_time_us":125}
Apr 29 14:05:02 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.195@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:05:02 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.195@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:05:02.077248 12735 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:02.070428 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.195:50925)
0429 14:05:02.070810 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0429 14:05:02.070828 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0429 14:05:02.071083 (+ 255us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:05:02.071290 (+ 207us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:05:02.071290 (+ 0us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:05:02.071499 (+ 209us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:05:02.071924 (+ 425us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:02.071932 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:02.073074 (+ 1142us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:05:02.073076 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:05:02.073484 (+ 408us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:02.073486 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:02.073791 (+ 305us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:02.074693 (+ 902us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:05:02.074699 (+ 6us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:05:02.077161 (+ 2462us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":158,"threads_started":1}
W20260429 14:05:02.077328 12732 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:05:02.070519 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:50925 (local address 127.2.197.254:37333)
0429 14:05:02.070841 (+ 322us) server_negotiation.cc:207] Beginning negotiation
0429 14:05:02.070843 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:05:02.071057 (+ 214us) server_negotiation.cc:408] Connection header received
0429 14:05:02.071191 (+ 134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:05:02.071192 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:05:02.071205 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:05:02.071282 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:05:02.072114 (+ 832us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:02.072933 (+ 819us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:05:02.073785 (+ 852us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:02.074014 (+ 229us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:02.077266 (+ 3252us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.195:50925: BlockingRecv error: recv got EOF from 127.2.197.195:50925 (error 108)
Metrics: {"server-negotiator.queue_time_us":295}
Apr 29 14:05:02 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:05:02 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:05:02.838773 12736 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:02.832256 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.193:46141)
0429 14:05:02.832427 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0429 14:05:02.832440 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 14:05:02.832595 (+ 155us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:05:02.832979 (+ 384us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:05:02.832981 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:05:02.833435 (+ 454us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:05:02.833728 (+ 293us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:02.833733 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:02.834607 (+ 874us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:05:02.834608 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:05:02.835050 (+ 442us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:02.835053 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:02.835130 (+ 77us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:02.836106 (+ 976us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:05:02.836112 (+ 6us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:05:02.838702 (+ 2590us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":89,"threads_started":1}
W20260429 14:05:02.839030 12737 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:05:02.832352 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:46141 (local address 127.2.197.254:37333)
0429 14:05:02.832478 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0429 14:05:02.832480 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:05:02.832517 (+ 37us) server_negotiation.cc:408] Connection header received
0429 14:05:02.832797 (+ 280us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:05:02.832800 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:05:02.832824 (+ 24us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:05:02.832927 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:05:02.833898 (+ 971us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:02.834480 (+ 582us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:05:02.835189 (+ 709us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:02.835534 (+ 345us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:02.838966 (+ 3432us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.193:46141: BlockingRecv error: recv got EOF from 127.2.197.193:46141 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":54,"threads_started":1}
Apr 29 14:05:02 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:05:02 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:05:02.957729 12738 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:02.951668 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.194:36479)
0429 14:05:02.951966 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0429 14:05:02.951980 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 14:05:02.952050 (+ 70us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:05:02.952428 (+ 378us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:05:02.952429 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:05:02.952597 (+ 168us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:05:02.952760 (+ 163us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:02.952762 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:02.953729 (+ 967us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:05:02.953731 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:05:02.954259 (+ 528us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:02.954261 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:02.954736 (+ 475us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:02.955291 (+ 555us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:05:02.955296 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:05:02.957656 (+ 2360us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":112,"threads_started":1}
W20260429 14:05:02.957870 12737 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:05:02.952133 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:36479 (local address 127.2.197.254:37333)
0429 14:05:02.952245 (+ 112us) server_negotiation.cc:207] Beginning negotiation
0429 14:05:02.952247 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:05:02.952265 (+ 18us) server_negotiation.cc:408] Connection header received
0429 14:05:02.952300 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:05:02.952301 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:05:02.952313 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:05:02.952379 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:05:02.952937 (+ 558us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:02.953546 (+ 609us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:05:02.954782 (+ 1236us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:02.954927 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:02.957776 (+ 2849us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.194:36479: BlockingRecv error: recv got EOF from 127.2.197.194:36479 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Apr 29 14:05:03 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.195@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:05:03 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.195@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:05:03.082849 12739 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:03.077883 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.195:56139)
0429 14:05:03.078105 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0429 14:05:03.078116 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 14:05:03.078213 (+ 97us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:05:03.078552 (+ 339us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:05:03.078553 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:05:03.078763 (+ 210us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:05:03.078935 (+ 172us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:03.078938 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:03.079633 (+ 695us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:05:03.079634 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:05:03.080027 (+ 393us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:03.080029 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:03.080121 (+ 92us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:03.080763 (+ 642us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:05:03.080768 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:05:03.082764 (+ 1996us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":106,"threads_started":1}
W20260429 14:05:03.082937 12737 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:05:03.077996 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:56139 (local address 127.2.197.254:37333)
0429 14:05:03.078094 (+ 98us) server_negotiation.cc:207] Beginning negotiation
0429 14:05:03.078096 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:05:03.078193 (+ 97us) server_negotiation.cc:408] Connection header received
0429 14:05:03.078428 (+ 235us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:05:03.078430 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:05:03.078443 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:05:03.078542 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:05:03.079028 (+ 486us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:03.079510 (+ 482us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:05:03.080119 (+ 609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:03.080300 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:03.082875 (+ 2575us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.195:56139: BlockingRecv error: recv got EOF from 127.2.197.195:56139 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Apr 29 14:05:03 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:05:03 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:05:03.843750 12740 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:03.839354 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.193:33915)
0429 14:05:03.839532 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0429 14:05:03.839544 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:05:03.839603 (+ 59us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:05:03.839840 (+ 237us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:05:03.839842 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:05:03.840006 (+ 164us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:05:03.840171 (+ 165us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:03.840173 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:03.840889 (+ 716us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:05:03.840891 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:05:03.841260 (+ 369us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:03.841262 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:03.841328 (+ 66us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:03.841841 (+ 513us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:05:03.841846 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:05:03.843660 (+ 1814us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":109,"threads_started":1}
W20260429 14:05:03.843818 12741 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:05:03.839489 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:33915 (local address 127.2.197.254:37333)
0429 14:05:03.839667 (+ 178us) server_negotiation.cc:207] Beginning negotiation
0429 14:05:03.839669 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:05:03.839677 (+ 8us) server_negotiation.cc:408] Connection header received
0429 14:05:03.839699 (+ 22us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:05:03.839700 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:05:03.839709 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:05:03.839772 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:05:03.840249 (+ 477us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:03.840790 (+ 541us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:05:03.841349 (+ 559us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:03.841507 (+ 158us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:03.843755 (+ 2248us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.193:33915: BlockingRecv error: recv got EOF from 127.2.197.193:33915 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":124,"threads_started":1}
Apr 29 14:05:03 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:05:03 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:05:03.963037 12742 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:03.958452 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.194:48279)
0429 14:05:03.958645 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0429 14:05:03.958658 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 14:05:03.958711 (+ 53us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:05:03.958854 (+ 143us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:05:03.958855 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:05:03.959014 (+ 159us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:05:03.959185 (+ 171us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:03.959187 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:03.959896 (+ 709us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:05:03.959897 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:05:03.960276 (+ 379us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:03.960277 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:03.960337 (+ 60us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:03.960878 (+ 541us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:05:03.960882 (+ 4us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:05:03.962977 (+ 2095us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":99,"threads_started":1}
W20260429 14:05:03.963186 12741 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:05:03.958598 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:48279 (local address 127.2.197.254:37333)
0429 14:05:03.958688 (+ 90us) server_negotiation.cc:207] Beginning negotiation
0429 14:05:03.958690 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:05:03.958729 (+ 39us) server_negotiation.cc:408] Connection header received
0429 14:05:03.958763 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:05:03.958764 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:05:03.958776 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:05:03.958828 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:05:03.959287 (+ 459us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:03.959808 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:05:03.960353 (+ 545us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:03.960473 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:03.963108 (+ 2635us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.194:48279: BlockingRecv error: recv got EOF from 127.2.197.194:48279 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Apr 29 14:05:04 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.195@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:05:04 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.195@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:05:04.088213 12743 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:04.083515 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.195:45841)
0429 14:05:04.083676 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0429 14:05:04.083695 (+ 19us) client_negotiation.cc:175] Beginning negotiation
0429 14:05:04.083756 (+ 61us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:05:04.083949 (+ 193us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:05:04.083950 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:05:04.084121 (+ 171us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:05:04.084296 (+ 175us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:04.084298 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:04.084985 (+ 687us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:05:04.084987 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:05:04.085361 (+ 374us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:04.085362 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:04.085430 (+ 68us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:04.086097 (+ 667us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:05:04.086102 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:05:04.088152 (+ 2050us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":99,"threads_started":1}
W20260429 14:05:04.088656 12741 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:05:04.083618 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:45841 (local address 127.2.197.254:37333)
0429 14:05:04.083683 (+ 65us) server_negotiation.cc:207] Beginning negotiation
0429 14:05:04.083685 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:05:04.083785 (+ 100us) server_negotiation.cc:408] Connection header received
0429 14:05:04.083816 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:05:04.083817 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:05:04.083828 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:05:04.083891 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:05:04.084391 (+ 500us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:04.084894 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:05:04.085469 (+ 575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:04.085613 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:04.088601 (+ 2988us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.195:45841: BlockingRecv error: recv got EOF from 127.2.197.195:45841 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Apr 29 14:05:04 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:05:04 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.193@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:05:04.849357 12744 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:04.844457 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.193:59191)
0429 14:05:04.844689 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0429 14:05:04.844701 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:05:04.844760 (+ 59us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:05:04.844937 (+ 177us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:05:04.844938 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:05:04.845280 (+ 342us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:05:04.845499 (+ 219us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:04.845502 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:04.846266 (+ 764us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:05:04.846267 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:05:04.846712 (+ 445us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:04.846714 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:04.846805 (+ 91us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:04.847323 (+ 518us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:05:04.847338 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:05:04.849252 (+ 1914us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":83,"threads_started":1}
W20260429 14:05:04.849412 12745 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:05:04.844609 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:59191 (local address 127.2.197.254:37333)
0429 14:05:04.844743 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0429 14:05:04.844756 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0429 14:05:04.844814 (+ 58us) server_negotiation.cc:408] Connection header received
0429 14:05:04.844847 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:05:04.844849 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:05:04.844862 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:05:04.844932 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:05:04.845617 (+ 685us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:04.846159 (+ 542us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:05:04.846833 (+ 674us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:04.846973 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:04.849346 (+ 2373us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.193:59191: BlockingRecv error: recv got EOF from 127.2.197.193:59191 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":49,"threads_started":1}
Apr 29 14:05:04 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 14:05:04 dist-test-slave-8n77 krb5kdc[11668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.197.194@KRBTEST.COM for oryx/127.2.197.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 14:05:04.968816 12746 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:04.963649 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:37333 (local address 127.2.197.194:46667)
0429 14:05:04.963962 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0429 14:05:04.963975 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 14:05:04.964080 (+ 105us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:05:04.964435 (+ 355us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:05:04.964437 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:05:04.964665 (+ 228us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:05:04.964823 (+ 158us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:04.964825 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:04.965501 (+ 676us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:05:04.965502 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:05:04.965894 (+ 392us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:04.965896 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:04.965985 (+ 89us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:04.966588 (+ 603us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:05:04.966593 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:05:04.968701 (+ 2108us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.197.254:37333: Server oryx/127.2.197.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":213,"threads_started":1}
W20260429 14:05:04.968943 12745 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:05:04.963861 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:46667 (local address 127.2.197.254:37333)
0429 14:05:04.963941 (+ 80us) server_negotiation.cc:207] Beginning negotiation
0429 14:05:04.963944 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:05:04.964058 (+ 114us) server_negotiation.cc:408] Connection header received
0429 14:05:04.964324 (+ 266us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:05:04.964325 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:05:04.964347 (+ 22us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:05:04.964410 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:05:04.964925 (+ 515us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:04.965407 (+ 482us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:05:04.965978 (+ 571us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:04.966163 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:04.968852 (+ 2689us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.197.194:46667: BlockingRecv error: recv got EOF from 127.2.197.194:46667 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260429 14:05:05.001485 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 12265
I20260429 14:05:05.006459 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 12419
I20260429 14:05:05.011270 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 12552
I20260429 14:05:05.015996 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 12194
2026-04-29T14:05:05Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18591 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-04-29T14:05:05Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:05:05Z Disabled control of system clock
I20260429 14:05:05.044713 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:38327
--webserver_interface=127.2.197.254
--webserver_port=0
--builtin_ntp_servers=127.2.197.212:45343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:38327
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260429 14:05:05.125248 12750 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:05:05.125442 12750 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:05:05.125463 12750 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:05:05.127132 12750 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:05:05.127187 12750 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:05:05.127202 12750 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:05:05.127213 12750 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:05:05.127226 12750 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:05:05.128898 12750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:45343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:38327
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:38327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:05:05.129174 12750 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:05:05.129439 12750 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260429 14:05:05.132455 12750 server_base.cc:1061] running on GCE node
W20260429 14:05:05.132609 12758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:05:05.132704 12755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:05:05.132786 12756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:05:05.133025 12750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:05:05.133296 12750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:05:05.134465 12750 hybrid_clock.cc:648] HybridClock initialized: now 1777471505134454 us; error 34 us; skew 500 ppm
I20260429 14:05:05.135699 12750 webserver.cc:492] Webserver started at http://127.2.197.254:42897/ using document root <none> and password file <none>
I20260429 14:05:05.135926 12750 fs_manager.cc:362] Metadata directory not provided
I20260429 14:05:05.135977 12750 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:05:05.136085 12750 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:05:05.137032 12750 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "b9abb28054ef47e0b5932f1340c3fdda"
format_stamp: "Formatted at 2026-04-29 14:05:05 on dist-test-slave-8n77"
server_key: "6b44ee333a8c148088818c350f9ea050"
server_key_iv: "41b8bf19137c30d93cfa569148b75391"
server_key_version: "encryptionkey@0"
I20260429 14:05:05.137367 12750 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "b9abb28054ef47e0b5932f1340c3fdda"
format_stamp: "Formatted at 2026-04-29 14:05:05 on dist-test-slave-8n77"
server_key: "6b44ee333a8c148088818c350f9ea050"
server_key_iv: "41b8bf19137c30d93cfa569148b75391"
server_key_version: "encryptionkey@0"
I20260429 14:05:05.138681 12750 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260429 14:05:05.139617 12764 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:05:05.139848 12750 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:05:05.139940 12750 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "b9abb28054ef47e0b5932f1340c3fdda"
format_stamp: "Formatted at 2026-04-29 14:05:05 on dist-test-slave-8n77"
server_key: "6b44ee333a8c148088818c350f9ea050"
server_key_iv: "41b8bf19137c30d93cfa569148b75391"
server_key_version: "encryptionkey@0"
I20260429 14:05:05.140023 12750 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:05:05.155599 12750 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:05:05.155948 12750 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:05:05.156100 12750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:05:05.160318 12750 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:38327
I20260429 14:05:05.160367 12816 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:38327 every 8 connection(s)
I20260429 14:05:05.160703 12750 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:05:05.161319 12817 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:05:05.163820 12817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9abb28054ef47e0b5932f1340c3fdda: Bootstrap starting.
I20260429 14:05:05.164458 12817 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9abb28054ef47e0b5932f1340c3fdda: Neither blocks nor log segments found. Creating new log.
I20260429 14:05:05.164745 12817 log.cc:826] T 00000000000000000000000000000000 P b9abb28054ef47e0b5932f1340c3fdda: Log is configured to *not* fsync() on all Append() calls
I20260429 14:05:05.165501 12817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9abb28054ef47e0b5932f1340c3fdda: No bootstrap required, opened a new log
I20260429 14:05:05.166852 12817 raft_consensus.cc:359] T 00000000000000000000000000000000 P b9abb28054ef47e0b5932f1340c3fdda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9abb28054ef47e0b5932f1340c3fdda" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 38327 } }
I20260429 14:05:05.166973 12817 raft_consensus.cc:385] T 00000000000000000000000000000000 P b9abb28054ef47e0b5932f1340c3fdda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:05:05.166994 12817 raft_consensus.cc:740] T 00000000000000000000000000000000 P b9abb28054ef47e0b5932f1340c3fdda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9abb28054ef47e0b5932f1340c3fdda, State: Initialized, Role: FOLLOWER
I20260429 14:05:05.167129 12817 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9abb28054ef47e0b5932f1340c3fdda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9abb28054ef47e0b5932f1340c3fdda" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 38327 } }
I20260429 14:05:05.167198 12817 raft_consensus.cc:399] T 00000000000000000000000000000000 P b9abb28054ef47e0b5932f1340c3fdda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:05:05.167220 12817 raft_consensus.cc:493] T 00000000000000000000000000000000 P b9abb28054ef47e0b5932f1340c3fdda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:05:05.167246 12817 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b9abb28054ef47e0b5932f1340c3fdda [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:05:05.167829 12817 raft_consensus.cc:515] T 00000000000000000000000000000000 P b9abb28054ef47e0b5932f1340c3fdda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9abb28054ef47e0b5932f1340c3fdda" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 38327 } }
I20260429 14:05:05.167953 12817 leader_election.cc:304] T 00000000000000000000000000000000 P b9abb28054ef47e0b5932f1340c3fdda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9abb28054ef47e0b5932f1340c3fdda; no voters:
I20260429 14:05:05.168138 12817 leader_election.cc:290] T 00000000000000000000000000000000 P b9abb28054ef47e0b5932f1340c3fdda [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:05:05.168303 12821 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b9abb28054ef47e0b5932f1340c3fdda [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:05:05.168366 12817 sys_catalog.cc:565] T 00000000000000000000000000000000 P b9abb28054ef47e0b5932f1340c3fdda [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:05:05.168423 12821 raft_consensus.cc:697] T 00000000000000000000000000000000 P b9abb28054ef47e0b5932f1340c3fdda [term 1 LEADER]: Becoming Leader. State: Replica: b9abb28054ef47e0b5932f1340c3fdda, State: Running, Role: LEADER
I20260429 14:05:05.168495 12821 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9abb28054ef47e0b5932f1340c3fdda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9abb28054ef47e0b5932f1340c3fdda" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 38327 } }
I20260429 14:05:05.168877 12822 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9abb28054ef47e0b5932f1340c3fdda [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9abb28054ef47e0b5932f1340c3fdda. Latest consensus state: current_term: 1 leader_uuid: "b9abb28054ef47e0b5932f1340c3fdda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9abb28054ef47e0b5932f1340c3fdda" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 38327 } } }
I20260429 14:05:05.168926 12820 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9abb28054ef47e0b5932f1340c3fdda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9abb28054ef47e0b5932f1340c3fdda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9abb28054ef47e0b5932f1340c3fdda" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 38327 } } }
I20260429 14:05:05.169045 12822 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9abb28054ef47e0b5932f1340c3fdda [sys.catalog]: This master's current role is: LEADER
I20260429 14:05:05.169064 12820 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9abb28054ef47e0b5932f1340c3fdda [sys.catalog]: This master's current role is: LEADER
I20260429 14:05:05.169436 12830 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:05:05.170222 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 12750
I20260429 14:05:05.170460 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:05:05.170539 12830 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:05:05.170888 2839 external_mini_cluster.cc:1468] Setting key 416ec41910a63eaaa2aba61f25b48a7a
I20260429 14:05:05.173576 12830 catalog_manager.cc:1357] Generated new cluster ID: 2caf44b0272e40f08d173dcfc9b457f2
I20260429 14:05:05.173666 12830 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:05:05.177033 12839 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:05.171745 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54636 (local address 127.2.197.254:38327)
0429 14:05:05.171928 (+ 183us) server_negotiation.cc:207] Beginning negotiation
0429 14:05:05.171935 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 14:05:05.172202 (+ 267us) server_negotiation.cc:408] Connection header received
0429 14:05:05.172362 (+ 160us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:05:05.172371 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:05:05.172444 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:05:05.172613 (+ 169us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:05:05.173113 (+ 500us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:05.174301 (+ 1188us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:05:05.175309 (+ 1008us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:05.175668 (+ 359us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:05.176185 (+ 517us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:05:05.176192 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:05:05.176193 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:05:05.176461 (+ 268us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:05:05.176488 (+ 27us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:05:05.176496 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:05:05.176582 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0429 14:05:05.176786 (+ 204us) server_negotiation.cc:300] Negotiation successful
0429 14:05:05.176850 (+ 64us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":92,"threads_started":1}
I20260429 14:05:05.192444 12830 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:05:05.193235 12830 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:05:05.203244 12830 catalog_manager.cc:6044] T 00000000000000000000000000000000 P b9abb28054ef47e0b5932f1340c3fdda: Generated new TSK 0
I20260429 14:05:05.203447 12830 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 14:05:05.213243 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.193:0
--local_ip_for_outbound_sockets=127.2.197.193
--webserver_interface=127.2.197.193
--webserver_port=0
--tserver_master_addrs=127.2.197.254:38327
--builtin_ntp_servers=127.2.197.212:45343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260429 14:05:05.297153 12842 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:05:05.297406 12842 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:05:05.297446 12842 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:05:05.299237 12842 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:05:05.299340 12842 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:05:05.299386 12842 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.193
I20260429 14:05:05.301201 12842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:45343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.197.193
--webserver_port=0
--tserver_master_addrs=127.2.197.254:38327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.193
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:05:05.301537 12842 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:05:05.301812 12842 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:05:05.304998 12847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:05:05.304991 12848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:05:05.304991 12850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:05:05.305898 12842 server_base.cc:1061] running on GCE node
I20260429 14:05:05.306135 12842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:05:05.306412 12842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:05:05.307605 12842 hybrid_clock.cc:648] HybridClock initialized: now 1777471505307578 us; error 41 us; skew 500 ppm
I20260429 14:05:05.308873 12842 webserver.cc:492] Webserver started at http://127.2.197.193:36847/ using document root <none> and password file <none>
I20260429 14:05:05.309092 12842 fs_manager.cc:362] Metadata directory not provided
I20260429 14:05:05.309161 12842 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:05:05.309269 12842 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:05:05.310241 12842 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/data/instance:
uuid: "b61d223fc0934547b6814d994a50601f"
format_stamp: "Formatted at 2026-04-29 14:05:05 on dist-test-slave-8n77"
server_key: "c29b3dc2e5576137ea18612e190a898e"
server_key_iv: "adccda78539a9f2dfa538cf45f06837a"
server_key_version: "encryptionkey@0"
I20260429 14:05:05.310814 12842 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance:
uuid: "b61d223fc0934547b6814d994a50601f"
format_stamp: "Formatted at 2026-04-29 14:05:05 on dist-test-slave-8n77"
server_key: "c29b3dc2e5576137ea18612e190a898e"
server_key_iv: "adccda78539a9f2dfa538cf45f06837a"
server_key_version: "encryptionkey@0"
I20260429 14:05:05.312265 12842 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260429 14:05:05.313292 12856 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:05:05.313545 12842 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:05:05.313644 12842 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/wal
uuid: "b61d223fc0934547b6814d994a50601f"
format_stamp: "Formatted at 2026-04-29 14:05:05 on dist-test-slave-8n77"
server_key: "c29b3dc2e5576137ea18612e190a898e"
server_key_iv: "adccda78539a9f2dfa538cf45f06837a"
server_key_version: "encryptionkey@0"
I20260429 14:05:05.313726 12842 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:05:05.329540 12842 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:05:05.329897 12842 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:05:05.330052 12842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:05:05.330335 12842 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:05:05.330713 12842 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:05:05.330775 12842 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:05:05.330822 12842 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:05:05.330853 12842 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:05:05.337494 12842 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.193:44423
I20260429 14:05:05.337572 12970 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.193:44423 every 8 connection(s)
I20260429 14:05:05.337909 12842 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
I20260429 14:05:05.338469 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 12842
I20260429 14:05:05.338605 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance
I20260429 14:05:05.338970 2839 external_mini_cluster.cc:1468] Setting key e8b117e8cf7d4b1dc0324b043320a3a4
I20260429 14:05:05.340077 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.194:0
--local_ip_for_outbound_sockets=127.2.197.194
--webserver_interface=127.2.197.194
--webserver_port=0
--tserver_master_addrs=127.2.197.254:38327
--builtin_ntp_servers=127.2.197.212:45343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260429 14:05:05.343144 12839 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:05.338927 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:49489 (local address 127.2.197.254:38327)
0429 14:05:05.338965 (+ 38us) server_negotiation.cc:207] Beginning negotiation
0429 14:05:05.338967 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:05:05.339110 (+ 143us) server_negotiation.cc:408] Connection header received
0429 14:05:05.339921 (+ 811us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:05:05.339923 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:05:05.339935 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:05:05.339995 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:05:05.340551 (+ 556us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:05.341310 (+ 759us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:05:05.342098 (+ 788us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:05.342358 (+ 260us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:05.342806 (+ 448us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:05:05.342810 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:05:05.342810 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:05:05.342815 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:05:05.342831 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:05:05.342835 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:05:05.342893 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
0429 14:05:05.342988 (+ 95us) server_negotiation.cc:300] Negotiation successful
0429 14:05:05.343025 (+ 37us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260429 14:05:05.343506 12973 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:05.338684 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:38327 (local address 127.2.197.193:49489)
0429 14:05:05.338923 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0429 14:05:05.338959 (+ 36us) client_negotiation.cc:175] Beginning negotiation
0429 14:05:05.339826 (+ 867us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:05:05.340049 (+ 223us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:05:05.340053 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:05:05.340064 (+ 11us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:05:05.340450 (+ 386us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:05.340455 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:05.341409 (+ 954us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:05:05.341410 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:05:05.342018 (+ 608us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:05.342021 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:05.342146 (+ 125us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:05.342679 (+ 533us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 14:05:05.342680 (+ 1us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:05:05.342693 (+ 13us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 14:05:05.342715 (+ 22us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:05:05.342903 (+ 188us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:05:05.342906 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:05:05.342909 (+ 3us) client_negotiation.cc:770] Sending connection context
0429 14:05:05.342975 (+ 66us) client_negotiation.cc:241] Negotiation successful
0429 14:05:05.343085 (+ 110us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":73,"threads_started":1}
I20260429 14:05:05.344168 12971 heartbeater.cc:344] Connected to a master server at 127.2.197.254:38327
I20260429 14:05:05.344252 12971 heartbeater.cc:461] Registering TS with master...
I20260429 14:05:05.344444 12971 heartbeater.cc:507] Master 127.2.197.254:38327 requested a full tablet report, sending...
I20260429 14:05:05.344950 12781 ts_manager.cc:194] Registered new tserver with Master: b61d223fc0934547b6814d994a50601f (127.2.197.193:44423)
I20260429 14:05:05.345732 12781 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.2.197.193:49489
W20260429 14:05:05.423143 12974 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:05:05.423326 12974 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:05:05.423347 12974 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:05:05.424880 12974 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:05:05.424929 12974 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:05:05.424988 12974 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.194
I20260429 14:05:05.426791 12974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:45343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.197.194
--webserver_port=0
--tserver_master_addrs=127.2.197.254:38327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.194
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:05:05.427032 12974 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:05:05.427222 12974 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:05:05.430095 12979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:05:05.430083 12980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:05:05.430119 12982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:05:05.430476 12974 server_base.cc:1061] running on GCE node
I20260429 14:05:05.430687 12974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:05:05.430919 12974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:05:05.432087 12974 hybrid_clock.cc:648] HybridClock initialized: now 1777471505432056 us; error 48 us; skew 500 ppm
I20260429 14:05:05.433259 12974 webserver.cc:492] Webserver started at http://127.2.197.194:42407/ using document root <none> and password file <none>
I20260429 14:05:05.433543 12974 fs_manager.cc:362] Metadata directory not provided
I20260429 14:05:05.433619 12974 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:05:05.433784 12974 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:05:05.434810 12974 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/data/instance:
uuid: "040d4f1cb3384b189a9b4e3487fb2b4b"
format_stamp: "Formatted at 2026-04-29 14:05:05 on dist-test-slave-8n77"
server_key: "d6358d134aac9d1b9a84ff45a49a9e94"
server_key_iv: "4f0df1bec6e98f63eabfce9cc3fb36e6"
server_key_version: "encryptionkey@0"
I20260429 14:05:05.435132 12974 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance:
uuid: "040d4f1cb3384b189a9b4e3487fb2b4b"
format_stamp: "Formatted at 2026-04-29 14:05:05 on dist-test-slave-8n77"
server_key: "d6358d134aac9d1b9a84ff45a49a9e94"
server_key_iv: "4f0df1bec6e98f63eabfce9cc3fb36e6"
server_key_version: "encryptionkey@0"
I20260429 14:05:05.436374 12974 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260429 14:05:05.437229 12988 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:05:05.437485 12974 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:05:05.437580 12974 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/wal
uuid: "040d4f1cb3384b189a9b4e3487fb2b4b"
format_stamp: "Formatted at 2026-04-29 14:05:05 on dist-test-slave-8n77"
server_key: "d6358d134aac9d1b9a84ff45a49a9e94"
server_key_iv: "4f0df1bec6e98f63eabfce9cc3fb36e6"
server_key_version: "encryptionkey@0"
I20260429 14:05:05.437660 12974 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:05:05.446123 12974 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:05:05.446480 12974 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:05:05.446627 12974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:05:05.446849 12974 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:05:05.447163 12974 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:05:05.447217 12974 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:05:05.447261 12974 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:05:05.447299 12974 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:05:05.452836 12974 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.194:38343
I20260429 14:05:05.452927 13102 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.194:38343 every 8 connection(s)
I20260429 14:05:05.453213 12974 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
I20260429 14:05:05.455765 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 12974
I20260429 14:05:05.455919 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance
I20260429 14:05:05.456161 2839 external_mini_cluster.cc:1468] Setting key fc1fa7396086b731b0aed56f8eb0b4be
I20260429 14:05:05.457373 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.195:0
--local_ip_for_outbound_sockets=127.2.197.195
--webserver_interface=127.2.197.195
--webserver_port=0
--tserver_master_addrs=127.2.197.254:38327
--builtin_ntp_servers=127.2.197.212:45343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260429 14:05:05.459297 12839 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:05.453971 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:44403 (local address 127.2.197.254:38327)
0429 14:05:05.454079 (+ 108us) server_negotiation.cc:207] Beginning negotiation
0429 14:05:05.454082 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:05:05.454249 (+ 167us) server_negotiation.cc:408] Connection header received
0429 14:05:05.454876 (+ 627us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:05:05.454877 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:05:05.454896 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:05:05.454943 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:05:05.455889 (+ 946us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:05.456592 (+ 703us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:05:05.457588 (+ 996us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:05.458314 (+ 726us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:05.458860 (+ 546us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:05:05.458864 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:05:05.458865 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:05:05.458870 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:05:05.458884 (+ 14us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:05:05.458888 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:05:05.458942 (+ 54us) server_negotiation.cc:1036] Waiting for connection context
0429 14:05:05.459149 (+ 207us) server_negotiation.cc:300] Negotiation successful
0429 14:05:05.459213 (+ 64us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260429 14:05:05.459390 13105 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:05.453939 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:38327 (local address 127.2.197.194:44403)
0429 14:05:05.454137 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0429 14:05:05.454172 (+ 35us) client_negotiation.cc:175] Beginning negotiation
0429 14:05:05.454701 (+ 529us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:05:05.455003 (+ 302us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:05:05.455007 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:05:05.455022 (+ 15us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:05:05.455301 (+ 279us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:05.455303 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:05.456695 (+ 1392us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:05:05.456696 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:05:05.457400 (+ 704us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:05.457405 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:05.457595 (+ 190us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:05.458706 (+ 1111us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 14:05:05.458707 (+ 1us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:05:05.458724 (+ 17us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 14:05:05.458751 (+ 27us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:05:05.458964 (+ 213us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:05:05.458967 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:05:05.458971 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 14:05:05.459055 (+ 84us) client_negotiation.cc:241] Negotiation successful
0429 14:05:05.459164 (+ 109us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":80,"threads_started":1}
I20260429 14:05:05.460162 13103 heartbeater.cc:344] Connected to a master server at 127.2.197.254:38327
I20260429 14:05:05.460263 13103 heartbeater.cc:461] Registering TS with master...
I20260429 14:05:05.460469 13103 heartbeater.cc:507] Master 127.2.197.254:38327 requested a full tablet report, sending...
I20260429 14:05:05.460912 12781 ts_manager.cc:194] Registered new tserver with Master: 040d4f1cb3384b189a9b4e3487fb2b4b (127.2.197.194:38343)
I20260429 14:05:05.461287 12781 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.2.197.194:44403
W20260429 14:05:05.537685 13106 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:05:05.537901 13106 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:05:05.537933 13106 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:05:05.539579 13106 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:05:05.539652 13106 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:05:05.539693 13106 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.195
I20260429 14:05:05.541301 13106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:45343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.197.195
--webserver_port=0
--tserver_master_addrs=127.2.197.254:38327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.195
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:05:05.541672 13106 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:05:05.541965 13106 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:05:05.544625 13112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:05:05.544613 13111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:05:05.544612 13114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:05:05.545383 13106 server_base.cc:1061] running on GCE node
I20260429 14:05:05.545581 13106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:05:05.545805 13106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:05:05.546957 13106 hybrid_clock.cc:648] HybridClock initialized: now 1777471505546940 us; error 39 us; skew 500 ppm
I20260429 14:05:05.548135 13106 webserver.cc:492] Webserver started at http://127.2.197.195:44569/ using document root <none> and password file <none>
I20260429 14:05:05.548362 13106 fs_manager.cc:362] Metadata directory not provided
I20260429 14:05:05.548429 13106 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:05:05.548542 13106 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:05:05.549484 13106 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/data/instance:
uuid: "e674815631d04ee482be930c6ab0f64d"
format_stamp: "Formatted at 2026-04-29 14:05:05 on dist-test-slave-8n77"
server_key: "6d2a52719f207df8a1a1814602197374"
server_key_iv: "55ca77976efe6cf972edeb87e3fc59a9"
server_key_version: "encryptionkey@0"
I20260429 14:05:05.549809 13106 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance:
uuid: "e674815631d04ee482be930c6ab0f64d"
format_stamp: "Formatted at 2026-04-29 14:05:05 on dist-test-slave-8n77"
server_key: "6d2a52719f207df8a1a1814602197374"
server_key_iv: "55ca77976efe6cf972edeb87e3fc59a9"
server_key_version: "encryptionkey@0"
I20260429 14:05:05.551054 13106 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260429 14:05:05.551821 13120 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:05:05.551991 13106 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260429 14:05:05.552066 13106 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/wal
uuid: "e674815631d04ee482be930c6ab0f64d"
format_stamp: "Formatted at 2026-04-29 14:05:05 on dist-test-slave-8n77"
server_key: "6d2a52719f207df8a1a1814602197374"
server_key_iv: "55ca77976efe6cf972edeb87e3fc59a9"
server_key_version: "encryptionkey@0"
I20260429 14:05:05.552140 13106 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:05:05.570235 13106 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:05:05.570611 13106 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:05:05.570750 13106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:05:05.570969 13106 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:05:05.571285 13106 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:05:05.571337 13106 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:05:05.571378 13106 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:05:05.571424 13106 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:05:05.579646 13106 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.195:46169
I20260429 14:05:05.579677 13233 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.195:46169 every 8 connection(s)
I20260429 14:05:05.579998 13106 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
I20260429 14:05:05.584142 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 13106
I20260429 14:05:05.584215 12839 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:05.580834 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:33701 (local address 127.2.197.254:38327)
0429 14:05:05.580911 (+ 77us) server_negotiation.cc:207] Beginning negotiation
0429 14:05:05.580913 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:05:05.581149 (+ 236us) server_negotiation.cc:408] Connection header received
0429 14:05:05.581601 (+ 452us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:05:05.581603 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:05:05.581613 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:05:05.581678 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:05:05.582105 (+ 427us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:05.582601 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:05:05.583247 (+ 646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:05.583403 (+ 156us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:05.583892 (+ 489us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:05:05.583895 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:05:05.583896 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:05:05.583900 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:05:05.583947 (+ 47us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:05:05.583952 (+ 5us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:05:05.584015 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
0429 14:05:05.584079 (+ 64us) server_negotiation.cc:300] Negotiation successful
0429 14:05:05.584118 (+ 39us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260429 14:05:05.584262 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance
I20260429 14:05:05.584313 13236 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:05.580823 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:38327 (local address 127.2.197.195:33701)
0429 14:05:05.581014 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0429 14:05:05.581046 (+ 32us) client_negotiation.cc:175] Beginning negotiation
0429 14:05:05.581499 (+ 453us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:05:05.581737 (+ 238us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:05:05.581740 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:05:05.581748 (+ 8us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:05:05.582034 (+ 286us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:05.582036 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:05.582681 (+ 645us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:05:05.582682 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:05:05.583074 (+ 392us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:05.583076 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:05.583288 (+ 212us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:05.583780 (+ 492us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 14:05:05.583781 (+ 1us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:05:05.583791 (+ 10us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 14:05:05.583809 (+ 18us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:05:05.584021 (+ 212us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:05:05.584023 (+ 2us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:05:05.584025 (+ 2us) client_negotiation.cc:770] Sending connection context
0429 14:05:05.584078 (+ 53us) client_negotiation.cc:241] Negotiation successful
0429 14:05:05.584137 (+ 59us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":100,"threads_started":1}
I20260429 14:05:05.584451 2839 external_mini_cluster.cc:1468] Setting key 4700785bb50a57d28b8bab6c2833595e
I20260429 14:05:05.585114 13234 heartbeater.cc:344] Connected to a master server at 127.2.197.254:38327
I20260429 14:05:05.585229 13234 heartbeater.cc:461] Registering TS with master...
I20260429 14:05:05.585448 13234 heartbeater.cc:507] Master 127.2.197.254:38327 requested a full tablet report, sending...
I20260429 14:05:05.585846 12780 ts_manager.cc:194] Registered new tserver with Master: e674815631d04ee482be930c6ab0f64d (127.2.197.195:46169)
I20260429 14:05:05.586252 12780 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.2.197.195:33701
I20260429 14:05:05.596583 2839 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260429 14:05:05.598843 12839 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:05:05.598233 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54640 (local address 127.2.197.254:38327)
0429 14:05:05.598314 (+ 81us) server_negotiation.cc:207] Beginning negotiation
0429 14:05:05.598317 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:05:05.598407 (+ 90us) server_negotiation.cc:408] Connection header received
0429 14:05:05.598446 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:05:05.598447 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:05:05.598457 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:05:05.598502 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:05:05.598752 (+ 250us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54640: BlockingRecv error: recv got EOF from 127.0.0.1:54640 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260429 14:05:05.599001 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 12842
I20260429 14:05:05.603826 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 12974
I20260429 14:05:05.608508 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 13106
I20260429 14:05:05.613688 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 12750
2026-04-29T14:05:05Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (597 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-04-29T14:05:05Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:05:05Z Disabled control of system clock
I20260429 14:05:05.642211 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:32779
--webserver_interface=127.2.197.254
--webserver_port=0
--builtin_ntp_servers=127.2.197.212:34487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:32779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260429 14:05:05.720713 13245 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:05:05.720921 13245 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:05:05.720947 13245 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:05:05.722543 13245 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:05:05.722599 13245 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:05:05.722615 13245 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:05:05.722628 13245 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:05:05.722642 13245 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:05:05.724320 13245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:34487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:32779
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:32779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:05:05.724591 13245 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:05:05.724848 13245 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:05:05.727697 13251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:05:05.727787 13250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:05:05.727697 13253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:05:05.728379 13245 server_base.cc:1061] running on GCE node
I20260429 14:05:05.728652 13245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:05:05.728920 13245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:05:05.730075 13245 hybrid_clock.cc:648] HybridClock initialized: now 1777471505730047 us; error 49 us; skew 500 ppm
I20260429 14:05:05.731283 13245 webserver.cc:492] Webserver started at http://127.2.197.254:35413/ using document root <none> and password file <none>
I20260429 14:05:05.731503 13245 fs_manager.cc:362] Metadata directory not provided
I20260429 14:05:05.731571 13245 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:05:05.731678 13245 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:05:05.732640 13245 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "eeca54ed528e423ab3b8b5ac3e36c527"
format_stamp: "Formatted at 2026-04-29 14:05:05 on dist-test-slave-8n77"
server_key: "5b8a3a1e78844a2964197f13d39b50c0"
server_key_iv: "af05ab2edcdd074912484ac95c21694f"
server_key_version: "encryptionkey@0"
I20260429 14:05:05.732972 13245 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "eeca54ed528e423ab3b8b5ac3e36c527"
format_stamp: "Formatted at 2026-04-29 14:05:05 on dist-test-slave-8n77"
server_key: "5b8a3a1e78844a2964197f13d39b50c0"
server_key_iv: "af05ab2edcdd074912484ac95c21694f"
server_key_version: "encryptionkey@0"
I20260429 14:05:05.734663 13245 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260429 14:05:05.735996 13259 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:05:05.736259 13245 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:05:05.736351 13245 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "eeca54ed528e423ab3b8b5ac3e36c527"
format_stamp: "Formatted at 2026-04-29 14:05:05 on dist-test-slave-8n77"
server_key: "5b8a3a1e78844a2964197f13d39b50c0"
server_key_iv: "af05ab2edcdd074912484ac95c21694f"
server_key_version: "encryptionkey@0"
I20260429 14:05:05.736436 13245 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:05:05.739830 13245 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:05:05.740097 13245 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:05:05.740229 13245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:05:05.743922 13245 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:32779
I20260429 14:05:05.743988 13313 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:32779 every 8 connection(s)
I20260429 14:05:05.744305 13245 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:05:05.744899 13314 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:05:05.746855 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 13245
I20260429 14:05:05.746964 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:05:05.747117 2839 external_mini_cluster.cc:1468] Setting key 71a0103452ae60034e335539f9b17aea
I20260429 14:05:05.747665 13314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eeca54ed528e423ab3b8b5ac3e36c527: Bootstrap starting.
I20260429 14:05:05.748306 13314 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eeca54ed528e423ab3b8b5ac3e36c527: Neither blocks nor log segments found. Creating new log.
I20260429 14:05:05.748631 13314 log.cc:826] T 00000000000000000000000000000000 P eeca54ed528e423ab3b8b5ac3e36c527: Log is configured to *not* fsync() on all Append() calls
I20260429 14:05:05.749586 13314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eeca54ed528e423ab3b8b5ac3e36c527: No bootstrap required, opened a new log
I20260429 14:05:05.749608 13317 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:05.747700 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36038 (local address 127.2.197.254:32779)
0429 14:05:05.747910 (+ 210us) server_negotiation.cc:207] Beginning negotiation
0429 14:05:05.747915 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 14:05:05.747933 (+ 18us) server_negotiation.cc:408] Connection header received
0429 14:05:05.748241 (+ 308us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:05:05.748246 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:05:05.748277 (+ 31us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:05:05.748379 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:05:05.749149 (+ 770us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:05:05.749152 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:05:05.749153 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:05:05.749158 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:05:05.749184 (+ 26us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:05:05.749191 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:05:05.749268 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0429 14:05:05.749363 (+ 95us) server_negotiation.cc:300] Negotiation successful
0429 14:05:05.749426 (+ 63us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":89,"threads_started":1}
I20260429 14:05:05.750928 13314 raft_consensus.cc:359] T 00000000000000000000000000000000 P eeca54ed528e423ab3b8b5ac3e36c527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeca54ed528e423ab3b8b5ac3e36c527" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 32779 } }
I20260429 14:05:05.751065 13314 raft_consensus.cc:385] T 00000000000000000000000000000000 P eeca54ed528e423ab3b8b5ac3e36c527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:05:05.751111 13314 raft_consensus.cc:740] T 00000000000000000000000000000000 P eeca54ed528e423ab3b8b5ac3e36c527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeca54ed528e423ab3b8b5ac3e36c527, State: Initialized, Role: FOLLOWER
I20260429 14:05:05.751227 13314 consensus_queue.cc:260] T 00000000000000000000000000000000 P eeca54ed528e423ab3b8b5ac3e36c527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeca54ed528e423ab3b8b5ac3e36c527" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 32779 } }
I20260429 14:05:05.751310 13314 raft_consensus.cc:399] T 00000000000000000000000000000000 P eeca54ed528e423ab3b8b5ac3e36c527 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:05:05.751351 13314 raft_consensus.cc:493] T 00000000000000000000000000000000 P eeca54ed528e423ab3b8b5ac3e36c527 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:05:05.751397 13314 raft_consensus.cc:3060] T 00000000000000000000000000000000 P eeca54ed528e423ab3b8b5ac3e36c527 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:05:05.752539 13314 raft_consensus.cc:515] T 00000000000000000000000000000000 P eeca54ed528e423ab3b8b5ac3e36c527 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeca54ed528e423ab3b8b5ac3e36c527" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 32779 } }
I20260429 14:05:05.752694 13314 leader_election.cc:304] T 00000000000000000000000000000000 P eeca54ed528e423ab3b8b5ac3e36c527 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eeca54ed528e423ab3b8b5ac3e36c527; no voters:
I20260429 14:05:05.752889 13314 leader_election.cc:290] T 00000000000000000000000000000000 P eeca54ed528e423ab3b8b5ac3e36c527 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:05:05.752938 13319 raft_consensus.cc:2804] T 00000000000000000000000000000000 P eeca54ed528e423ab3b8b5ac3e36c527 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:05:05.753118 13319 raft_consensus.cc:697] T 00000000000000000000000000000000 P eeca54ed528e423ab3b8b5ac3e36c527 [term 1 LEADER]: Becoming Leader. State: Replica: eeca54ed528e423ab3b8b5ac3e36c527, State: Running, Role: LEADER
I20260429 14:05:05.753166 13314 sys_catalog.cc:565] T 00000000000000000000000000000000 P eeca54ed528e423ab3b8b5ac3e36c527 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:05:05.753199 13319 consensus_queue.cc:237] T 00000000000000000000000000000000 P eeca54ed528e423ab3b8b5ac3e36c527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeca54ed528e423ab3b8b5ac3e36c527" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 32779 } }
I20260429 14:05:05.753671 13320 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeca54ed528e423ab3b8b5ac3e36c527 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eeca54ed528e423ab3b8b5ac3e36c527" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeca54ed528e423ab3b8b5ac3e36c527" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 32779 } } }
I20260429 14:05:05.753679 13321 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeca54ed528e423ab3b8b5ac3e36c527 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eeca54ed528e423ab3b8b5ac3e36c527. Latest consensus state: current_term: 1 leader_uuid: "eeca54ed528e423ab3b8b5ac3e36c527" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeca54ed528e423ab3b8b5ac3e36c527" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 32779 } } }
I20260429 14:05:05.753850 13321 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeca54ed528e423ab3b8b5ac3e36c527 [sys.catalog]: This master's current role is: LEADER
I20260429 14:05:05.753804 13320 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeca54ed528e423ab3b8b5ac3e36c527 [sys.catalog]: This master's current role is: LEADER
I20260429 14:05:05.754558 13328 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:05:05.755112 13328 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:05:05.756479 13328 catalog_manager.cc:1357] Generated new cluster ID: 3ffd38068da549a799f3bd6ca7a52416
I20260429 14:05:05.756539 13328 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:05:05.771837 13328 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:05:05.772034 13328 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:05:05.780174 13328 catalog_manager.cc:6044] T 00000000000000000000000000000000 P eeca54ed528e423ab3b8b5ac3e36c527: Generated new TSK 0
I20260429 14:05:05.780383 13328 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 14:05:05.789692 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.193:0
--local_ip_for_outbound_sockets=127.2.197.193
--webserver_interface=127.2.197.193
--webserver_port=0
--tserver_master_addrs=127.2.197.254:32779
--builtin_ntp_servers=127.2.197.212:34487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260429 14:05:05.893091 13339 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:05:05.893302 13339 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:05:05.893338 13339 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:05:05.894922 13339 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:05:05.894995 13339 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:05:05.895040 13339 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.193
I20260429 14:05:05.896661 13339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:34487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.197.193
--webserver_port=0
--tserver_master_addrs=127.2.197.254:32779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.193
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:05:05.896935 13339 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:05:05.897164 13339 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:05:05.899730 13344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:05:05.899804 13347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:05:05.899842 13345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:05:05.900068 13339 server_base.cc:1061] running on GCE node
I20260429 14:05:05.900238 13339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:05:05.900440 13339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:05:05.901597 13339 hybrid_clock.cc:648] HybridClock initialized: now 1777471505901583 us; error 24 us; skew 500 ppm
I20260429 14:05:05.902922 13339 webserver.cc:492] Webserver started at http://127.2.197.193:33837/ using document root <none> and password file <none>
I20260429 14:05:05.903142 13339 fs_manager.cc:362] Metadata directory not provided
I20260429 14:05:05.903187 13339 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:05:05.903293 13339 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:05:05.904234 13339 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/data/instance:
uuid: "be4d95f62b4d4c2baa5260cdbd3869f0"
format_stamp: "Formatted at 2026-04-29 14:05:05 on dist-test-slave-8n77"
server_key: "6e699ab784cc618127a6bc41876221c2"
server_key_iv: "7a42a02599ce8a1238df9451c57410ba"
server_key_version: "encryptionkey@0"
I20260429 14:05:05.904567 13339 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance:
uuid: "be4d95f62b4d4c2baa5260cdbd3869f0"
format_stamp: "Formatted at 2026-04-29 14:05:05 on dist-test-slave-8n77"
server_key: "6e699ab784cc618127a6bc41876221c2"
server_key_iv: "7a42a02599ce8a1238df9451c57410ba"
server_key_version: "encryptionkey@0"
I20260429 14:05:05.905838 13339 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260429 14:05:05.906714 13353 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:05:05.906956 13339 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260429 14:05:05.907033 13339 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/wal
uuid: "be4d95f62b4d4c2baa5260cdbd3869f0"
format_stamp: "Formatted at 2026-04-29 14:05:05 on dist-test-slave-8n77"
server_key: "6e699ab784cc618127a6bc41876221c2"
server_key_iv: "7a42a02599ce8a1238df9451c57410ba"
server_key_version: "encryptionkey@0"
I20260429 14:05:05.907100 13339 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:05:05.910723 13339 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:05:05.911022 13339 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:05:05.911163 13339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:05:05.911374 13339 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:05:05.911711 13339 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:05:05.911764 13339 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:05:05.911800 13339 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:05:05.911842 13339 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:05:05.918017 13339 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.193:43859
I20260429 14:05:05.918135 13467 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.193:43859 every 8 connection(s)
I20260429 14:05:05.918488 13339 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
I20260429 14:05:05.921475 13317 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:05.919337 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:52397 (local address 127.2.197.254:32779)
0429 14:05:05.919414 (+ 77us) server_negotiation.cc:207] Beginning negotiation
0429 14:05:05.919417 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:05:05.919748 (+ 331us) server_negotiation.cc:408] Connection header received
0429 14:05:05.920214 (+ 466us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:05:05.920216 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:05:05.920223 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:05:05.920301 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:05:05.921165 (+ 864us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:05:05.921167 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:05:05.921167 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:05:05.921172 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:05:05.921191 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:05:05.921195 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:05:05.921266 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0429 14:05:05.921390 (+ 124us) server_negotiation.cc:300] Negotiation successful
0429 14:05:05.921407 (+ 17us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260429 14:05:05.921588 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 13339
I20260429 14:05:05.921639 13470 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:05.919303 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:32779 (local address 127.2.197.193:52397)
0429 14:05:05.919499 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0429 14:05:05.919532 (+ 33us) client_negotiation.cc:175] Beginning negotiation
0429 14:05:05.920118 (+ 586us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:05:05.920350 (+ 232us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:05:05.920353 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:05:05.920361 (+ 8us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:05:05.921033 (+ 672us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 14:05:05.921034 (+ 1us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:05:05.921054 (+ 20us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 14:05:05.921073 (+ 19us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:05:05.921287 (+ 214us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:05:05.921292 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:05:05.921295 (+ 3us) client_negotiation.cc:770] Sending connection context
0429 14:05:05.921406 (+ 111us) client_negotiation.cc:241] Negotiation successful
0429 14:05:05.921454 (+ 48us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":96,"threads_started":1}
I20260429 14:05:05.921708 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance
I20260429 14:05:05.921885 2839 external_mini_cluster.cc:1468] Setting key 4443b09daee64bab0d8c966bad480be8
I20260429 14:05:05.922670 13468 heartbeater.cc:344] Connected to a master server at 127.2.197.254:32779
I20260429 14:05:05.922813 13468 heartbeater.cc:461] Registering TS with master...
I20260429 14:05:05.922964 13468 heartbeater.cc:507] Master 127.2.197.254:32779 requested a full tablet report, sending...
I20260429 14:05:05.922998 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.194:0
--local_ip_for_outbound_sockets=127.2.197.194
--webserver_interface=127.2.197.194
--webserver_port=0
--tserver_master_addrs=127.2.197.254:32779
--builtin_ntp_servers=127.2.197.212:34487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260429 14:05:05.923534 13276 ts_manager.cc:194] Registered new tserver with Master: be4d95f62b4d4c2baa5260cdbd3869f0 (127.2.197.193:43859)
W20260429 14:05:06.002818 13471 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:05:06.003006 13471 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:05:06.003031 13471 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:05:06.004451 13471 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:05:06.004501 13471 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:05:06.004529 13471 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.194
I20260429 14:05:06.006047 13471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:34487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.197.194
--webserver_port=0
--tserver_master_addrs=127.2.197.254:32779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.194
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:05:06.006242 13471 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:05:06.006503 13471 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:05:06.009050 13476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:05:06.009151 13477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:05:06.009184 13479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:05:06.009254 13471 server_base.cc:1061] running on GCE node
I20260429 14:05:06.009541 13471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:05:06.009780 13471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:05:06.010917 13471 hybrid_clock.cc:648] HybridClock initialized: now 1777471506010898 us; error 40 us; skew 500 ppm
I20260429 14:05:06.012006 13471 webserver.cc:492] Webserver started at http://127.2.197.194:43657/ using document root <none> and password file <none>
I20260429 14:05:06.012214 13471 fs_manager.cc:362] Metadata directory not provided
I20260429 14:05:06.012279 13471 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:05:06.012383 13471 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:05:06.013353 13471 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/data/instance:
uuid: "dd6062285d5147d092238cc8ccb99e32"
format_stamp: "Formatted at 2026-04-29 14:05:06 on dist-test-slave-8n77"
server_key: "3c6b8b20312347f176cd9e704794230e"
server_key_iv: "96ef475e5693f2796529050833a7ab23"
server_key_version: "encryptionkey@0"
I20260429 14:05:06.013813 13471 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance:
uuid: "dd6062285d5147d092238cc8ccb99e32"
format_stamp: "Formatted at 2026-04-29 14:05:06 on dist-test-slave-8n77"
server_key: "3c6b8b20312347f176cd9e704794230e"
server_key_iv: "96ef475e5693f2796529050833a7ab23"
server_key_version: "encryptionkey@0"
I20260429 14:05:06.015177 13471 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260429 14:05:06.016105 13485 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:05:06.016314 13471 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:05:06.016407 13471 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/wal
uuid: "dd6062285d5147d092238cc8ccb99e32"
format_stamp: "Formatted at 2026-04-29 14:05:06 on dist-test-slave-8n77"
server_key: "3c6b8b20312347f176cd9e704794230e"
server_key_iv: "96ef475e5693f2796529050833a7ab23"
server_key_version: "encryptionkey@0"
I20260429 14:05:06.016486 13471 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:05:06.020365 13471 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:05:06.020638 13471 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:05:06.020778 13471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:05:06.020992 13471 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:05:06.021297 13471 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:05:06.021353 13471 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:05:06.021395 13471 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:05:06.021420 13471 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:05:06.027348 13471 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.194:36367
I20260429 14:05:06.027506 13599 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.194:36367 every 8 connection(s)
I20260429 14:05:06.027779 13471 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
I20260429 14:05:06.030669 13317 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:06.028786 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:54163 (local address 127.2.197.254:32779)
0429 14:05:06.028870 (+ 84us) server_negotiation.cc:207] Beginning negotiation
0429 14:05:06.028872 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:05:06.029074 (+ 202us) server_negotiation.cc:408] Connection header received
0429 14:05:06.029524 (+ 450us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:05:06.029526 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:05:06.029538 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:05:06.029600 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:05:06.030336 (+ 736us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:05:06.030338 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:05:06.030338 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:05:06.030346 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:05:06.030359 (+ 13us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:05:06.030363 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:05:06.030414 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
0429 14:05:06.030602 (+ 188us) server_negotiation.cc:300] Negotiation successful
0429 14:05:06.030616 (+ 14us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260429 14:05:06.030814 13602 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:06.028702 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:32779 (local address 127.2.197.194:54163)
0429 14:05:06.028886 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0429 14:05:06.028919 (+ 33us) client_negotiation.cc:175] Beginning negotiation
0429 14:05:06.029432 (+ 513us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:05:06.029659 (+ 227us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:05:06.029663 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:05:06.029670 (+ 7us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:05:06.030173 (+ 503us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 14:05:06.030173 (+ 0us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:05:06.030183 (+ 10us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 14:05:06.030201 (+ 18us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:05:06.030505 (+ 304us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:05:06.030510 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:05:06.030513 (+ 3us) client_negotiation.cc:770] Sending connection context
0429 14:05:06.030598 (+ 85us) client_negotiation.cc:241] Negotiation successful
0429 14:05:06.030633 (+ 35us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":105,"threads_started":1}
I20260429 14:05:06.031361 13600 heartbeater.cc:344] Connected to a master server at 127.2.197.254:32779
I20260429 14:05:06.031495 13600 heartbeater.cc:461] Registering TS with master...
I20260429 14:05:06.031631 13600 heartbeater.cc:507] Master 127.2.197.254:32779 requested a full tablet report, sending...
I20260429 14:05:06.032130 13276 ts_manager.cc:194] Registered new tserver with Master: dd6062285d5147d092238cc8ccb99e32 (127.2.197.194:36367)
I20260429 14:05:06.037814 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 13471
I20260429 14:05:06.037914 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance
I20260429 14:05:06.038123 2839 external_mini_cluster.cc:1468] Setting key 1641a10a1b096ddb5ce7b45a6dbe0924
I20260429 14:05:06.039417 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.195:0
--local_ip_for_outbound_sockets=127.2.197.195
--webserver_interface=127.2.197.195
--webserver_port=0
--tserver_master_addrs=127.2.197.254:32779
--builtin_ntp_servers=127.2.197.212:34487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260429 14:05:06.119340 13603 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:05:06.119593 13603 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:05:06.119633 13603 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:05:06.121557 13603 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:05:06.121639 13603 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:05:06.121687 13603 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.195
I20260429 14:05:06.123529 13603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:34487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.197.195
--webserver_port=0
--tserver_master_addrs=127.2.197.254:32779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.195
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:05:06.123855 13603 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:05:06.124131 13603 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:05:06.126996 13609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:05:06.127004 13611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:05:06.127004 13608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:05:06.127384 13603 server_base.cc:1061] running on GCE node
I20260429 14:05:06.127548 13603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:05:06.127823 13603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:05:06.128990 13603 hybrid_clock.cc:648] HybridClock initialized: now 1777471506128957 us; error 54 us; skew 500 ppm
I20260429 14:05:06.130350 13603 webserver.cc:492] Webserver started at http://127.2.197.195:45243/ using document root <none> and password file <none>
I20260429 14:05:06.130594 13603 fs_manager.cc:362] Metadata directory not provided
I20260429 14:05:06.130659 13603 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:05:06.130774 13603 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:05:06.131812 13603 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/data/instance:
uuid: "417cd2e56a0f4e3495c26cb39427f717"
format_stamp: "Formatted at 2026-04-29 14:05:06 on dist-test-slave-8n77"
server_key: "43c86bf6e713d70a40bd8bb4a98810a8"
server_key_iv: "bec168e0141760ded6ac6a634b79b7e9"
server_key_version: "encryptionkey@0"
I20260429 14:05:06.132164 13603 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance:
uuid: "417cd2e56a0f4e3495c26cb39427f717"
format_stamp: "Formatted at 2026-04-29 14:05:06 on dist-test-slave-8n77"
server_key: "43c86bf6e713d70a40bd8bb4a98810a8"
server_key_iv: "bec168e0141760ded6ac6a634b79b7e9"
server_key_version: "encryptionkey@0"
I20260429 14:05:06.133455 13603 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260429 14:05:06.134423 13617 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:05:06.134716 13603 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:05:06.134811 13603 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/wal
uuid: "417cd2e56a0f4e3495c26cb39427f717"
format_stamp: "Formatted at 2026-04-29 14:05:06 on dist-test-slave-8n77"
server_key: "43c86bf6e713d70a40bd8bb4a98810a8"
server_key_iv: "bec168e0141760ded6ac6a634b79b7e9"
server_key_version: "encryptionkey@0"
I20260429 14:05:06.134898 13603 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:05:06.138831 13603 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:05:06.139119 13603 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:05:06.139262 13603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:05:06.139467 13603 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:05:06.139793 13603 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:05:06.139853 13603 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:05:06.139902 13603 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:05:06.139936 13603 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:05:06.146723 13603 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.195:41365
I20260429 14:05:06.146838 13731 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.195:41365 every 8 connection(s)
I20260429 14:05:06.147320 13603 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
I20260429 14:05:06.150426 13317 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:06.148208 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:56767 (local address 127.2.197.254:32779)
0429 14:05:06.148283 (+ 75us) server_negotiation.cc:207] Beginning negotiation
0429 14:05:06.148285 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:05:06.148593 (+ 308us) server_negotiation.cc:408] Connection header received
0429 14:05:06.149019 (+ 426us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:05:06.149021 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:05:06.149026 (+ 5us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:05:06.149092 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:05:06.149995 (+ 903us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:05:06.149997 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:05:06.149997 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:05:06.150002 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:05:06.150018 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:05:06.150022 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:05:06.150102 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0429 14:05:06.150270 (+ 168us) server_negotiation.cc:300] Negotiation successful
0429 14:05:06.150357 (+ 87us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260429 14:05:06.150832 13734 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:06.148203 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:32779 (local address 127.2.197.195:56767)
0429 14:05:06.148405 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0429 14:05:06.148432 (+ 27us) client_negotiation.cc:175] Beginning negotiation
0429 14:05:06.148897 (+ 465us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:05:06.149197 (+ 300us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:05:06.149201 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:05:06.149210 (+ 9us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:05:06.149803 (+ 593us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 14:05:06.149803 (+ 0us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:05:06.149821 (+ 18us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 14:05:06.149855 (+ 34us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:05:06.150155 (+ 300us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:05:06.150161 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:05:06.150165 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 14:05:06.150270 (+ 105us) client_negotiation.cc:241] Negotiation successful
0429 14:05:06.150473 (+ 203us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":99,"threads_started":1}
I20260429 14:05:06.151702 13732 heartbeater.cc:344] Connected to a master server at 127.2.197.254:32779
I20260429 14:05:06.151822 13732 heartbeater.cc:461] Registering TS with master...
I20260429 14:05:06.151921 13732 heartbeater.cc:507] Master 127.2.197.254:32779 requested a full tablet report, sending...
I20260429 14:05:06.152305 13276 ts_manager.cc:194] Registered new tserver with Master: 417cd2e56a0f4e3495c26cb39427f717 (127.2.197.195:41365)
I20260429 14:05:06.154098 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 13603
I20260429 14:05:06.154232 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance
I20260429 14:05:06.154467 2839 external_mini_cluster.cc:1468] Setting key 69e241dccd39fd206a97a19e83a23a82
I20260429 14:05:06.155506 2839 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260429 14:05:06.157567 13317 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:05:06.157064 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36054 (local address 127.2.197.254:32779)
0429 14:05:06.157133 (+ 69us) server_negotiation.cc:207] Beginning negotiation
0429 14:05:06.157135 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:05:06.157254 (+ 119us) server_negotiation.cc:408] Connection header received
0429 14:05:06.157293 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:05:06.157293 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:05:06.157298 (+ 5us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:05:06.157348 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:05:06.157508 (+ 160us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36054: BlockingRecv error: recv got EOF from 127.0.0.1:36054 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260429 14:05:06.157846 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 13339
I20260429 14:05:06.163300 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 13471
I20260429 14:05:06.168288 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 13603
I20260429 14:05:06.173019 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 13245
2026-04-29T14:05:06Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (557 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:05:06 dist-test-slave-8n77 krb5kdc[13742](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:05:06 dist-test-slave-8n77 krb5kdc[13742](info): set up 2 sockets
Apr 29 14:05:06 dist-test-slave-8n77 krb5kdc[13742](info): commencing operation
krb5kdc: starting...
W20260429 14:05:08.224278 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:05:08 dist-test-slave-8n77 krb5kdc[13742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:05:08Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:05:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:05:08.379141 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:35689
--webserver_interface=127.2.197.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:35689
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:05:08.460052 13758 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:05:08.460258 13758 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:05:08.460282 13758 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:05:08.461863 13758 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:05:08.461920 13758 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:05:08.461934 13758 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:05:08.461947 13758 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:05:08.461959 13758 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:05:08.463748 13758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:35689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:35689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:05:08.463964 13758 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:05:08.464103 13758 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:05:08.467217 13766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:05:08.467247 13764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:05:08.467272 13763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:05:08.467247 13758 server_base.cc:1061] running on GCE node
I20260429 14:05:08.467711 13758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:05:08.467937 13758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:05:08.469107 13758 hybrid_clock.cc:648] HybridClock initialized: now 1777471508469095 us; error 33 us; skew 500 ppm
Apr 29 14:05:08 dist-test-slave-8n77 krb5kdc[13742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471508, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:05:08.472143 13758 init.cc:377] Logged in from keytab as kudu/127.2.197.254@KRBTEST.COM (short username kudu)
I20260429 14:05:08.472781 13758 webserver.cc:492] Webserver started at http://127.2.197.254:33059/ using document root <none> and password file <none>
I20260429 14:05:08.472994 13758 fs_manager.cc:362] Metadata directory not provided
I20260429 14:05:08.473037 13758 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:05:08.473136 13758 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:05:08.474087 13758 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "b5504f4a0cfc430983fe26aa9478e57f"
format_stamp: "Formatted at 2026-04-29 14:05:08 on dist-test-slave-8n77"
server_key: "f2d2406ca37cb77bb4146713176e60b7"
server_key_iv: "9a56719af8c5a45dd27bc9a7c794c59f"
server_key_version: "encryptionkey@0"
I20260429 14:05:08.474447 13758 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "b5504f4a0cfc430983fe26aa9478e57f"
format_stamp: "Formatted at 2026-04-29 14:05:08 on dist-test-slave-8n77"
server_key: "f2d2406ca37cb77bb4146713176e60b7"
server_key_iv: "9a56719af8c5a45dd27bc9a7c794c59f"
server_key_version: "encryptionkey@0"
I20260429 14:05:08.475982 13758 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:05:08.477082 13773 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:05:08.477347 13758 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:05:08.477438 13758 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "b5504f4a0cfc430983fe26aa9478e57f"
format_stamp: "Formatted at 2026-04-29 14:05:08 on dist-test-slave-8n77"
server_key: "f2d2406ca37cb77bb4146713176e60b7"
server_key_iv: "9a56719af8c5a45dd27bc9a7c794c59f"
server_key_version: "encryptionkey@0"
I20260429 14:05:08.477523 13758 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:05:08.511502 13758 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:05:08.519858 13758 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:05:08.520056 13758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:05:08.524529 13758 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:35689
I20260429 14:05:08.524710 13825 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:35689 every 8 connection(s)
I20260429 14:05:08.524964 13758 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:05:08.525571 13826 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:05:08.528280 13826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5504f4a0cfc430983fe26aa9478e57f: Bootstrap starting.
I20260429 14:05:08.528855 13826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5504f4a0cfc430983fe26aa9478e57f: Neither blocks nor log segments found. Creating new log.
I20260429 14:05:08.529057 13826 log.cc:826] T 00000000000000000000000000000000 P b5504f4a0cfc430983fe26aa9478e57f: Log is configured to *not* fsync() on all Append() calls
I20260429 14:05:08.529812 13826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5504f4a0cfc430983fe26aa9478e57f: No bootstrap required, opened a new log
I20260429 14:05:08.531128 13826 raft_consensus.cc:359] T 00000000000000000000000000000000 P b5504f4a0cfc430983fe26aa9478e57f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5504f4a0cfc430983fe26aa9478e57f" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 35689 } }
I20260429 14:05:08.531242 13826 raft_consensus.cc:385] T 00000000000000000000000000000000 P b5504f4a0cfc430983fe26aa9478e57f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:05:08.531263 13826 raft_consensus.cc:740] T 00000000000000000000000000000000 P b5504f4a0cfc430983fe26aa9478e57f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5504f4a0cfc430983fe26aa9478e57f, State: Initialized, Role: FOLLOWER
I20260429 14:05:08.531384 13826 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5504f4a0cfc430983fe26aa9478e57f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5504f4a0cfc430983fe26aa9478e57f" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 35689 } }
I20260429 14:05:08.531455 13826 raft_consensus.cc:399] T 00000000000000000000000000000000 P b5504f4a0cfc430983fe26aa9478e57f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:05:08.531477 13826 raft_consensus.cc:493] T 00000000000000000000000000000000 P b5504f4a0cfc430983fe26aa9478e57f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:05:08.531502 13826 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b5504f4a0cfc430983fe26aa9478e57f [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:05:08.532008 13826 raft_consensus.cc:515] T 00000000000000000000000000000000 P b5504f4a0cfc430983fe26aa9478e57f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5504f4a0cfc430983fe26aa9478e57f" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 35689 } }
I20260429 14:05:08.532096 13826 leader_election.cc:304] T 00000000000000000000000000000000 P b5504f4a0cfc430983fe26aa9478e57f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5504f4a0cfc430983fe26aa9478e57f; no voters:
I20260429 14:05:08.532229 13826 leader_election.cc:290] T 00000000000000000000000000000000 P b5504f4a0cfc430983fe26aa9478e57f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:05:08.532320 13829 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b5504f4a0cfc430983fe26aa9478e57f [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:05:08.532490 13826 sys_catalog.cc:565] T 00000000000000000000000000000000 P b5504f4a0cfc430983fe26aa9478e57f [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:05:08.532533 13829 raft_consensus.cc:697] T 00000000000000000000000000000000 P b5504f4a0cfc430983fe26aa9478e57f [term 1 LEADER]: Becoming Leader. State: Replica: b5504f4a0cfc430983fe26aa9478e57f, State: Running, Role: LEADER
I20260429 14:05:08.532642 13829 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5504f4a0cfc430983fe26aa9478e57f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5504f4a0cfc430983fe26aa9478e57f" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 35689 } }
I20260429 14:05:08.532999 13830 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5504f4a0cfc430983fe26aa9478e57f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5504f4a0cfc430983fe26aa9478e57f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5504f4a0cfc430983fe26aa9478e57f" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 35689 } } }
I20260429 14:05:08.533010 13831 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5504f4a0cfc430983fe26aa9478e57f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5504f4a0cfc430983fe26aa9478e57f. Latest consensus state: current_term: 1 leader_uuid: "b5504f4a0cfc430983fe26aa9478e57f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5504f4a0cfc430983fe26aa9478e57f" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 35689 } } }
I20260429 14:05:08.533197 13830 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5504f4a0cfc430983fe26aa9478e57f [sys.catalog]: This master's current role is: LEADER
I20260429 14:05:08.533207 13831 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5504f4a0cfc430983fe26aa9478e57f [sys.catalog]: This master's current role is: LEADER
I20260429 14:05:08.533568 13838 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:05:08.534749 13838 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:05:08.535054 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 13758
I20260429 14:05:08.535161 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:05:08.535344 2839 external_mini_cluster.cc:1468] Setting key d8f86a4689569d519e3e4d393d444a9d
I20260429 14:05:08.537144 13838 catalog_manager.cc:1357] Generated new cluster ID: 059cf48e30ff43a2a382575a8edc6c61
I20260429 14:05:08.537261 13838 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Apr 29 14:05:08 dist-test-slave-8n77 krb5kdc[13742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:05:08.546665 13848 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:08.535983 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:50562 (local address 127.2.197.254:35689)
0429 14:05:08.536160 (+ 177us) server_negotiation.cc:207] Beginning negotiation
0429 14:05:08.536174 (+ 14us) server_negotiation.cc:400] Waiting for connection header
0429 14:05:08.536200 (+ 26us) server_negotiation.cc:408] Connection header received
0429 14:05:08.536310 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:05:08.536316 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:05:08.536387 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:05:08.536517 (+ 130us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:05:08.537668 (+ 1151us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:08.538946 (+ 1278us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:05:08.539786 (+ 840us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:08.540030 (+ 244us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:08.542777 (+ 2747us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:05:08.542785 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:05:08.542786 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:05:08.542804 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:05:08.545117 (+ 2313us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:05:08.545510 (+ 393us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:05:08.545513 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:05:08.545513 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:05:08.545581 (+ 68us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:05:08.545753 (+ 172us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:05:08.545754 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:05:08.545754 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:05:08.546023 (+ 269us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:05:08.546166 (+ 143us) server_negotiation.cc:1036] Waiting for connection context
0429 14:05:08.546310 (+ 144us) server_negotiation.cc:300] Negotiation successful
0429 14:05:08.546445 (+ 135us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":95,"threads_started":1}
I20260429 14:05:08.550714 13838 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:05:08.551384 13838 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:05:08.556128 13838 catalog_manager.cc:6044] T 00000000000000000000000000000000 P b5504f4a0cfc430983fe26aa9478e57f: Generated new TSK 0
I20260429 14:05:08.556322 13838 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:05:08.613762 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.193:0
--local_ip_for_outbound_sockets=127.2.197.193
--webserver_interface=127.2.197.193
--webserver_port=0
--tserver_master_addrs=127.2.197.254:35689
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:05:08.696694 13854 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:05:08.696957 13854 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:05:08.697000 13854 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:05:08.698666 13854 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:05:08.698755 13854 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:05:08.698807 13854 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.193
I20260429 14:05:08.700563 13854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.197.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:35689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.193
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:05:08.700862 13854 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:05:08.701071 13854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:05:08.703923 13860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:05:08.703931 13862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:05:08.704067 13859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:05:08.704406 13854 server_base.cc:1061] running on GCE node
I20260429 14:05:08.704586 13854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:05:08.704816 13854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:05:08.705988 13854 hybrid_clock.cc:648] HybridClock initialized: now 1777471508705971 us; error 28 us; skew 500 ppm
Apr 29 14:05:08 dist-test-slave-8n77 krb5kdc[13742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471508, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:05:08.709232 13854 init.cc:377] Logged in from keytab as kudu/127.2.197.193@KRBTEST.COM (short username kudu)
I20260429 14:05:08.709918 13854 webserver.cc:492] Webserver started at http://127.2.197.193:45431/ using document root <none> and password file <none>
I20260429 14:05:08.710129 13854 fs_manager.cc:362] Metadata directory not provided
I20260429 14:05:08.710170 13854 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:05:08.710330 13854 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:05:08.711282 13854 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/data/instance:
uuid: "5484942802e24c2291c633437870b3f6"
format_stamp: "Formatted at 2026-04-29 14:05:08 on dist-test-slave-8n77"
server_key: "e8964a1cf5dc023c7435512303dd7f4c"
server_key_iv: "2ec7978643093931008116e4080f6fdf"
server_key_version: "encryptionkey@0"
I20260429 14:05:08.711617 13854 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance:
uuid: "5484942802e24c2291c633437870b3f6"
format_stamp: "Formatted at 2026-04-29 14:05:08 on dist-test-slave-8n77"
server_key: "e8964a1cf5dc023c7435512303dd7f4c"
server_key_iv: "2ec7978643093931008116e4080f6fdf"
server_key_version: "encryptionkey@0"
I20260429 14:05:08.713217 13854 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260429 14:05:08.714368 13869 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:05:08.714592 13854 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:05:08.714665 13854 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/wal
uuid: "5484942802e24c2291c633437870b3f6"
format_stamp: "Formatted at 2026-04-29 14:05:08 on dist-test-slave-8n77"
server_key: "e8964a1cf5dc023c7435512303dd7f4c"
server_key_iv: "2ec7978643093931008116e4080f6fdf"
server_key_version: "encryptionkey@0"
I20260429 14:05:08.714879 13854 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:05:08.732529 13854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:05:08.735105 13854 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:05:08.735265 13854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:05:08.735838 13854 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:05:08.736258 13854 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:05:08.736325 13854 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:05:08.736368 13854 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:05:08.736399 13854 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:05:08.743116 13854 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.193:42027
I20260429 14:05:08.743211 13987 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.193:42027 every 8 connection(s)
I20260429 14:05:08.743506 13854 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
I20260429 14:05:08.749372 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 13854
I20260429 14:05:08.749468 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance
Apr 29 14:05:08 dist-test-slave-8n77 krb5kdc[13742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471508, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:05:08.749697 2839 external_mini_cluster.cc:1468] Setting key c2bc6036dff628165e1f7b0929f75566
I20260429 14:05:08.753839 13848 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:08.744494 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:34381 (local address 127.2.197.254:35689)
0429 14:05:08.744585 (+ 91us) server_negotiation.cc:207] Beginning negotiation
0429 14:05:08.744588 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:05:08.744732 (+ 144us) server_negotiation.cc:408] Connection header received
0429 14:05:08.745185 (+ 453us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:05:08.745186 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:05:08.745205 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:05:08.745275 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:05:08.746197 (+ 922us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:08.746797 (+ 600us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:05:08.747455 (+ 658us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:08.747614 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:08.750259 (+ 2645us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:05:08.750262 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:05:08.750263 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:05:08.750271 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:05:08.752024 (+ 1753us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:05:08.752804 (+ 780us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:05:08.752805 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:05:08.752806 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:05:08.752838 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:05:08.753099 (+ 261us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:05:08.753100 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:05:08.753101 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:05:08.753220 (+ 119us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:05:08.753305 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0429 14:05:08.753587 (+ 282us) server_negotiation.cc:300] Negotiation successful
0429 14:05:08.753724 (+ 137us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260429 14:05:08.753935 13990 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:08.744316 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:35689 (local address 127.2.197.193:34381)
0429 14:05:08.744592 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0429 14:05:08.744619 (+ 27us) client_negotiation.cc:175] Beginning negotiation
0429 14:05:08.745075 (+ 456us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:05:08.745361 (+ 286us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:05:08.745365 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:05:08.745771 (+ 406us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:05:08.746042 (+ 271us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:08.746046 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:08.746923 (+ 877us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:05:08.746924 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:05:08.747350 (+ 426us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:08.747352 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:08.747458 (+ 106us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:08.748434 (+ 976us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:05:08.748443 (+ 9us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:05:08.750164 (+ 1721us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:05:08.752284 (+ 2120us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:05:08.752299 (+ 15us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:05:08.752303 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:05:08.752658 (+ 355us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:05:08.752942 (+ 284us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:05:08.752943 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:05:08.752943 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:05:08.753015 (+ 72us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:05:08.753291 (+ 276us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:05:08.753296 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:05:08.753433 (+ 137us) client_negotiation.cc:770] Sending connection context
0429 14:05:08.753569 (+ 136us) client_negotiation.cc:241] Negotiation successful
0429 14:05:08.753729 (+ 160us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":125,"threads_started":1}
I20260429 14:05:08.754599 13988 heartbeater.cc:344] Connected to a master server at 127.2.197.254:35689
I20260429 14:05:08.754736 13988 heartbeater.cc:461] Registering TS with master...
I20260429 14:05:08.754951 13988 heartbeater.cc:507] Master 127.2.197.254:35689 requested a full tablet report, sending...
I20260429 14:05:08.755635 13790 ts_manager.cc:194] Registered new tserver with Master: 5484942802e24c2291c633437870b3f6 (127.2.197.193:42027)
I20260429 14:05:08.756292 13790 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.193@KRBTEST.COM'} at 127.2.197.193:34381
WARNING: no policy specified for kudu/127.2.197.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:05:08.806668 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.194:0
--local_ip_for_outbound_sockets=127.2.197.194
--webserver_interface=127.2.197.194
--webserver_port=0
--tserver_master_addrs=127.2.197.254:35689
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:05:08.889662 13995 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:05:08.889931 13995 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:05:08.889971 13995 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:05:08.891585 13995 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:05:08.891680 13995 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:05:08.891726 13995 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.194
I20260429 14:05:08.893488 13995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.197.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:35689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.194
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:05:08.893815 13995 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:05:08.894019 13995 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:05:08.897004 14000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:05:08.897083 14001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:05:08.897106 14003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:05:08.897341 13995 server_base.cc:1061] running on GCE node
I20260429 14:05:08.897547 13995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:05:08.897794 13995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:05:08.898948 13995 hybrid_clock.cc:648] HybridClock initialized: now 1777471508898933 us; error 27 us; skew 500 ppm
Apr 29 14:05:08 dist-test-slave-8n77 krb5kdc[13742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471508, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:05:08.901647 13995 init.cc:377] Logged in from keytab as kudu/127.2.197.194@KRBTEST.COM (short username kudu)
I20260429 14:05:08.902238 13995 webserver.cc:492] Webserver started at http://127.2.197.194:37397/ using document root <none> and password file <none>
I20260429 14:05:08.902467 13995 fs_manager.cc:362] Metadata directory not provided
I20260429 14:05:08.902529 13995 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:05:08.902640 13995 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:05:08.903596 13995 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/data/instance:
uuid: "5323c3aad1f64af6a392c1c1a59e3fa1"
format_stamp: "Formatted at 2026-04-29 14:05:08 on dist-test-slave-8n77"
server_key: "b16c7673eaea635c32e50f6de232b4b4"
server_key_iv: "8abad1a98acd438c56dd1a79c4ff25d7"
server_key_version: "encryptionkey@0"
I20260429 14:05:08.903936 13995 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance:
uuid: "5323c3aad1f64af6a392c1c1a59e3fa1"
format_stamp: "Formatted at 2026-04-29 14:05:08 on dist-test-slave-8n77"
server_key: "b16c7673eaea635c32e50f6de232b4b4"
server_key_iv: "8abad1a98acd438c56dd1a79c4ff25d7"
server_key_version: "encryptionkey@0"
I20260429 14:05:08.905242 13995 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260429 14:05:08.906105 14010 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:05:08.906380 13995 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:05:08.906462 13995 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/wal
uuid: "5323c3aad1f64af6a392c1c1a59e3fa1"
format_stamp: "Formatted at 2026-04-29 14:05:08 on dist-test-slave-8n77"
server_key: "b16c7673eaea635c32e50f6de232b4b4"
server_key_iv: "8abad1a98acd438c56dd1a79c4ff25d7"
server_key_version: "encryptionkey@0"
I20260429 14:05:08.906531 13995 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:05:08.919428 13995 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:05:08.921519 13995 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:05:08.921691 13995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:05:08.921952 13995 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:05:08.922365 13995 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:05:08.922420 13995 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:05:08.922461 13995 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:05:08.922489 13995 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:05:08.929394 13995 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.194:34339
I20260429 14:05:08.929495 14124 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.194:34339 every 8 connection(s)
I20260429 14:05:08.929816 13995 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
I20260429 14:05:08.931643 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 13995
I20260429 14:05:08.931725 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance
I20260429 14:05:08.931905 2839 external_mini_cluster.cc:1468] Setting key 9b465c59c0c0497618cf2547c8189e9e
Apr 29 14:05:08 dist-test-slave-8n77 krb5kdc[13742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471508, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.197.195@KRBTEST.COM; defaulting to no policy
I20260429 14:05:08.940117 13848 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:08.930680 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:40677 (local address 127.2.197.254:35689)
0429 14:05:08.930761 (+ 81us) server_negotiation.cc:207] Beginning negotiation
0429 14:05:08.930763 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:05:08.930916 (+ 153us) server_negotiation.cc:408] Connection header received
0429 14:05:08.931369 (+ 453us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:05:08.931370 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:05:08.931381 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:05:08.931477 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:05:08.932320 (+ 843us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:08.932834 (+ 514us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:05:08.933479 (+ 645us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:08.933652 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:08.936775 (+ 3123us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:05:08.936793 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:05:08.936793 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:05:08.936803 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:05:08.938729 (+ 1926us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:05:08.939214 (+ 485us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:05:08.939216 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:05:08.939216 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:05:08.939250 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:05:08.939456 (+ 206us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:05:08.939457 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:05:08.939457 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:05:08.939570 (+ 113us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:05:08.939652 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0429 14:05:08.939927 (+ 275us) server_negotiation.cc:300] Negotiation successful
0429 14:05:08.940021 (+ 94us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260429 14:05:08.940294 14127 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:08.930633 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:35689 (local address 127.2.197.194:40677)
0429 14:05:08.930809 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0429 14:05:08.930837 (+ 28us) client_negotiation.cc:175] Beginning negotiation
0429 14:05:08.931290 (+ 453us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:05:08.931520 (+ 230us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:05:08.931523 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:05:08.931887 (+ 364us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:05:08.932225 (+ 338us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:08.932230 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:08.932933 (+ 703us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:05:08.932935 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:05:08.933390 (+ 455us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:08.933392 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:08.933535 (+ 143us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:08.934612 (+ 1077us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:05:08.934622 (+ 10us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:05:08.936620 (+ 1998us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:05:08.938857 (+ 2237us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:05:08.938867 (+ 10us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:05:08.938870 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:05:08.939125 (+ 255us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:05:08.939326 (+ 201us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:05:08.939327 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:05:08.939327 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:05:08.939393 (+ 66us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:05:08.939641 (+ 248us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:05:08.939646 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:05:08.939784 (+ 138us) client_negotiation.cc:770] Sending connection context
0429 14:05:08.939909 (+ 125us) client_negotiation.cc:241] Negotiation successful
0429 14:05:08.940060 (+ 151us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":107,"threads_started":1}
I20260429 14:05:08.940932 14125 heartbeater.cc:344] Connected to a master server at 127.2.197.254:35689
I20260429 14:05:08.941064 14125 heartbeater.cc:461] Registering TS with master...
I20260429 14:05:08.941327 14125 heartbeater.cc:507] Master 127.2.197.254:35689 requested a full tablet report, sending...
I20260429 14:05:08.941951 13790 ts_manager.cc:194] Registered new tserver with Master: 5323c3aad1f64af6a392c1c1a59e3fa1 (127.2.197.194:34339)
I20260429 14:05:08.942446 13790 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.194@KRBTEST.COM'} at 127.2.197.194:40677
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:05:08.987305 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.195:0
--local_ip_for_outbound_sockets=127.2.197.195
--webserver_interface=127.2.197.195
--webserver_port=0
--tserver_master_addrs=127.2.197.254:35689
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:05:09.070164 14132 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:05:09.070497 14132 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:05:09.070528 14132 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:05:09.072919 14132 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:05:09.072991 14132 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:05:09.073024 14132 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.195
I20260429 14:05:09.074911 14132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.197.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:35689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.195
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:05:09.075117 14132 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:05:09.075261 14132 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:05:09.077960 14140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:05:09.077963 14138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:05:09.078233 14132 server_base.cc:1061] running on GCE node
W20260429 14:05:09.077991 14137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:05:09.078558 14132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:05:09.078794 14132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:05:09.079962 14132 hybrid_clock.cc:648] HybridClock initialized: now 1777471509079932 us; error 64 us; skew 500 ppm
Apr 29 14:05:09 dist-test-slave-8n77 krb5kdc[13742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471509, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:05:09.082839 14132 init.cc:377] Logged in from keytab as kudu/127.2.197.195@KRBTEST.COM (short username kudu)
I20260429 14:05:09.083436 14132 webserver.cc:492] Webserver started at http://127.2.197.195:46621/ using document root <none> and password file <none>
I20260429 14:05:09.083649 14132 fs_manager.cc:362] Metadata directory not provided
I20260429 14:05:09.083712 14132 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:05:09.083818 14132 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:05:09.084766 14132 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/data/instance:
uuid: "4294638a7919452cb23c37e9f0a30fcb"
format_stamp: "Formatted at 2026-04-29 14:05:09 on dist-test-slave-8n77"
server_key: "a37ae9585216ddbd186594d16950d7ce"
server_key_iv: "8697fb2e47d3942d1ec809c6838424c1"
server_key_version: "encryptionkey@0"
I20260429 14:05:09.085093 14132 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance:
uuid: "4294638a7919452cb23c37e9f0a30fcb"
format_stamp: "Formatted at 2026-04-29 14:05:09 on dist-test-slave-8n77"
server_key: "a37ae9585216ddbd186594d16950d7ce"
server_key_iv: "8697fb2e47d3942d1ec809c6838424c1"
server_key_version: "encryptionkey@0"
I20260429 14:05:09.086390 14132 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260429 14:05:09.087194 14147 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:05:09.087409 14132 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:05:09.087539 14132 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/wal
uuid: "4294638a7919452cb23c37e9f0a30fcb"
format_stamp: "Formatted at 2026-04-29 14:05:09 on dist-test-slave-8n77"
server_key: "a37ae9585216ddbd186594d16950d7ce"
server_key_iv: "8697fb2e47d3942d1ec809c6838424c1"
server_key_version: "encryptionkey@0"
I20260429 14:05:09.087627 14132 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:05:09.101692 14132 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:05:09.104167 14132 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:05:09.104352 14132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:05:09.104601 14132 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:05:09.104921 14132 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:05:09.104976 14132 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:05:09.105022 14132 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:05:09.105057 14132 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:05:09.111296 14132 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.195:38095
I20260429 14:05:09.111340 14261 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.195:38095 every 8 connection(s)
I20260429 14:05:09.111632 14132 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
I20260429 14:05:09.112169 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 14132
I20260429 14:05:09.112274 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance
I20260429 14:05:09.112448 2839 external_mini_cluster.cc:1468] Setting key 8950c372783cf797324fbefb437afde4
Apr 29 14:05:09 dist-test-slave-8n77 krb5kdc[13742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471509, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:05:09.121863 13848 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:09.112642 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:36151 (local address 127.2.197.254:35689)
0429 14:05:09.112706 (+ 64us) server_negotiation.cc:207] Beginning negotiation
0429 14:05:09.112708 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:05:09.112718 (+ 10us) server_negotiation.cc:408] Connection header received
0429 14:05:09.113223 (+ 505us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:05:09.113225 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:05:09.113237 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:05:09.113293 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:05:09.114055 (+ 762us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:09.114629 (+ 574us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:05:09.115187 (+ 558us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:09.115351 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:09.118018 (+ 2667us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:05:09.118023 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:05:09.118024 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:05:09.118034 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:05:09.119919 (+ 1885us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:05:09.120477 (+ 558us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:05:09.120478 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:05:09.120479 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:05:09.120515 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:05:09.120930 (+ 415us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:05:09.120932 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:05:09.120932 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:05:09.121062 (+ 130us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:05:09.121152 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0429 14:05:09.121461 (+ 309us) server_negotiation.cc:300] Negotiation successful
0429 14:05:09.121641 (+ 180us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260429 14:05:09.121986 14264 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:09.112353 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:35689 (local address 127.2.197.195:36151)
0429 14:05:09.112501 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0429 14:05:09.112533 (+ 32us) client_negotiation.cc:175] Beginning negotiation
0429 14:05:09.113122 (+ 589us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:05:09.113345 (+ 223us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:05:09.113350 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:05:09.113670 (+ 320us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:05:09.113953 (+ 283us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:09.113956 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:09.114729 (+ 773us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:05:09.114730 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:05:09.115104 (+ 374us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:09.115106 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:09.115236 (+ 130us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:09.116296 (+ 1060us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:05:09.116306 (+ 10us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:05:09.117908 (+ 1602us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:05:09.120079 (+ 2171us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:05:09.120093 (+ 14us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:05:09.120097 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:05:09.120382 (+ 285us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:05:09.120584 (+ 202us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:05:09.120585 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:05:09.120585 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:05:09.120674 (+ 89us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:05:09.121147 (+ 473us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:05:09.121158 (+ 11us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:05:09.121314 (+ 156us) client_negotiation.cc:770] Sending connection context
0429 14:05:09.121442 (+ 128us) client_negotiation.cc:241] Negotiation successful
0429 14:05:09.121607 (+ 165us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":85,"threads_started":1}
I20260429 14:05:09.122964 14262 heartbeater.cc:344] Connected to a master server at 127.2.197.254:35689
I20260429 14:05:09.123128 14262 heartbeater.cc:461] Registering TS with master...
I20260429 14:05:09.123482 14262 heartbeater.cc:507] Master 127.2.197.254:35689 requested a full tablet report, sending...
I20260429 14:05:09.124099 13790 ts_manager.cc:194] Registered new tserver with Master: 4294638a7919452cb23c37e9f0a30fcb (127.2.197.195:38095)
I20260429 14:05:09.124652 13790 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.195@KRBTEST.COM'} at 127.2.197.195:36151
I20260429 14:05:09.124658 2839 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260429 14:05:09.132866 13848 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:09.126150 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:50578 (local address 127.2.197.254:35689)
0429 14:05:09.126256 (+ 106us) server_negotiation.cc:207] Beginning negotiation
0429 14:05:09.126259 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:05:09.126386 (+ 127us) server_negotiation.cc:408] Connection header received
0429 14:05:09.126505 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:05:09.126506 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:05:09.126517 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:05:09.126592 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:05:09.127286 (+ 694us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:09.127797 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:05:09.128377 (+ 580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:09.128559 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:09.129747 (+ 1188us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:05:09.129752 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:05:09.129752 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:05:09.129759 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:05:09.131626 (+ 1867us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:05:09.132022 (+ 396us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:05:09.132023 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:05:09.132024 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:05:09.132055 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:05:09.132252 (+ 197us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:05:09.132253 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:05:09.132254 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:05:09.132389 (+ 135us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:05:09.132517 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
0429 14:05:09.132660 (+ 143us) server_negotiation.cc:300] Negotiation successful
0429 14:05:09.132750 (+ 90us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260429 14:05:09.134871 13790 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 14:05:09.136241 13790 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 14:05:09.142129 14273 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:09.138862 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:42027 (local address 127.0.0.1:35976)
0429 14:05:09.138971 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0429 14:05:09.139003 (+ 32us) client_negotiation.cc:175] Beginning negotiation
0429 14:05:09.139084 (+ 81us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:05:09.139434 (+ 350us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:05:09.139436 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:05:09.139439 (+ 3us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:05:09.139679 (+ 240us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:09.139681 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:09.140999 (+ 1318us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:05:09.141000 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:05:09.141826 (+ 826us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:09.141828 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:09.141920 (+ 92us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:09.141959 (+ 39us) client_negotiation.cc:770] Sending connection context
0429 14:05:09.142007 (+ 48us) client_negotiation.cc:241] Negotiation successful
0429 14:05:09.142049 (+ 42us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":63,"threads_started":1}
I20260429 14:05:09.142441 14277 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:09.139116 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:34339 (local address 127.0.0.1:47070)
0429 14:05:09.139602 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0429 14:05:09.139614 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:05:09.139672 (+ 58us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:05:09.139955 (+ 283us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:05:09.139956 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:05:09.139957 (+ 1us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:05:09.140153 (+ 196us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:09.140154 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:09.141128 (+ 974us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:05:09.141129 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:05:09.142155 (+ 1026us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:09.142157 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:09.142235 (+ 78us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:09.142240 (+ 5us) client_negotiation.cc:770] Sending connection context
0429 14:05:09.142309 (+ 69us) client_negotiation.cc:241] Negotiation successful
0429 14:05:09.142371 (+ 62us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":77,"threads_started":1}
I20260429 14:05:09.142696 14276 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:09.139018 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35976 (local address 127.2.197.193:42027)
0429 14:05:09.139262 (+ 244us) server_negotiation.cc:207] Beginning negotiation
0429 14:05:09.139270 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0429 14:05:09.139283 (+ 13us) server_negotiation.cc:408] Connection header received
0429 14:05:09.139325 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:05:09.139327 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:05:09.139347 (+ 20us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:05:09.139424 (+ 77us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:05:09.139777 (+ 353us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:09.140900 (+ 1123us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:05:09.141938 (+ 1038us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:09.142479 (+ 541us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:09.142537 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
0429 14:05:09.142568 (+ 31us) server_negotiation.cc:300] Negotiation successful
0429 14:05:09.142616 (+ 48us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":90,"threads_started":1}
I20260429 14:05:09.142720 14274 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:09.138868 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:38095 (local address 127.0.0.1:51544)
0429 14:05:09.139247 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0429 14:05:09.139259 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:05:09.139326 (+ 67us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:05:09.139782 (+ 456us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:05:09.139782 (+ 0us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:05:09.139785 (+ 3us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:05:09.140011 (+ 226us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:09.140012 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:09.141260 (+ 1248us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:05:09.141261 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:05:09.142175 (+ 914us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:09.142177 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:09.142254 (+ 77us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:09.142257 (+ 3us) client_negotiation.cc:770] Sending connection context
0429 14:05:09.142619 (+ 362us) client_negotiation.cc:241] Negotiation successful
0429 14:05:09.142661 (+ 42us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":105,"threads_started":1}
I20260429 14:05:09.143404 14278 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:09.139219 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:47070 (local address 127.2.197.194:34339)
0429 14:05:09.139373 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0429 14:05:09.139380 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 14:05:09.139757 (+ 377us) server_negotiation.cc:408] Connection header received
0429 14:05:09.139798 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:05:09.139800 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:05:09.139816 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:05:09.139880 (+ 64us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:05:09.140238 (+ 358us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:09.141032 (+ 794us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:05:09.142586 (+ 1554us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:09.143193 (+ 607us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:09.143237 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
0429 14:05:09.143270 (+ 33us) server_negotiation.cc:300] Negotiation successful
0429 14:05:09.143317 (+ 47us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":98,"threads_started":1}
I20260429 14:05:09.143502 14275 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:09.138932 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51544 (local address 127.2.197.195:38095)
0429 14:05:09.139204 (+ 272us) server_negotiation.cc:207] Beginning negotiation
0429 14:05:09.139213 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0429 14:05:09.139394 (+ 181us) server_negotiation.cc:408] Connection header received
0429 14:05:09.139433 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:05:09.139436 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:05:09.139508 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:05:09.139693 (+ 185us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:05:09.140094 (+ 401us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:09.141187 (+ 1093us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:05:09.142365 (+ 1178us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:09.143288 (+ 923us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:09.143323 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
0429 14:05:09.143350 (+ 27us) server_negotiation.cc:300] Negotiation successful
0429 14:05:09.143393 (+ 43us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":86,"threads_started":1}
I20260429 14:05:09.143393 13921 tablet_service.cc:1511] Processing CreateTablet for tablet 6fb9cad39c3844869e05df234c27cc9f (DEFAULT_TABLE table=test-table [id=ed3317dfeb614a0893bb6c34e082f505]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:05:09.143725 13921 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fb9cad39c3844869e05df234c27cc9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:05:09.144119 14196 tablet_service.cc:1511] Processing CreateTablet for tablet 6fb9cad39c3844869e05df234c27cc9f (DEFAULT_TABLE table=test-table [id=ed3317dfeb614a0893bb6c34e082f505]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:05:09.144377 14196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fb9cad39c3844869e05df234c27cc9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:05:09.146188 14279 tablet_bootstrap.cc:492] T 6fb9cad39c3844869e05df234c27cc9f P 5484942802e24c2291c633437870b3f6: Bootstrap starting.
I20260429 14:05:09.146642 14280 tablet_bootstrap.cc:492] T 6fb9cad39c3844869e05df234c27cc9f P 4294638a7919452cb23c37e9f0a30fcb: Bootstrap starting.
I20260429 14:05:09.146785 14059 tablet_service.cc:1511] Processing CreateTablet for tablet 6fb9cad39c3844869e05df234c27cc9f (DEFAULT_TABLE table=test-table [id=ed3317dfeb614a0893bb6c34e082f505]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:05:09.147059 14059 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fb9cad39c3844869e05df234c27cc9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:05:09.147117 14279 tablet_bootstrap.cc:654] T 6fb9cad39c3844869e05df234c27cc9f P 5484942802e24c2291c633437870b3f6: Neither blocks nor log segments found. Creating new log.
I20260429 14:05:09.147296 14280 tablet_bootstrap.cc:654] T 6fb9cad39c3844869e05df234c27cc9f P 4294638a7919452cb23c37e9f0a30fcb: Neither blocks nor log segments found. Creating new log.
I20260429 14:05:09.147480 14279 log.cc:826] T 6fb9cad39c3844869e05df234c27cc9f P 5484942802e24c2291c633437870b3f6: Log is configured to *not* fsync() on all Append() calls
I20260429 14:05:09.147617 14280 log.cc:826] T 6fb9cad39c3844869e05df234c27cc9f P 4294638a7919452cb23c37e9f0a30fcb: Log is configured to *not* fsync() on all Append() calls
I20260429 14:05:09.148345 14279 tablet_bootstrap.cc:492] T 6fb9cad39c3844869e05df234c27cc9f P 5484942802e24c2291c633437870b3f6: No bootstrap required, opened a new log
I20260429 14:05:09.148403 14279 ts_tablet_manager.cc:1403] T 6fb9cad39c3844869e05df234c27cc9f P 5484942802e24c2291c633437870b3f6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260429 14:05:09.149912 14283 tablet_bootstrap.cc:492] T 6fb9cad39c3844869e05df234c27cc9f P 5323c3aad1f64af6a392c1c1a59e3fa1: Bootstrap starting.
I20260429 14:05:09.150614 14283 tablet_bootstrap.cc:654] T 6fb9cad39c3844869e05df234c27cc9f P 5323c3aad1f64af6a392c1c1a59e3fa1: Neither blocks nor log segments found. Creating new log.
I20260429 14:05:09.150671 14280 tablet_bootstrap.cc:492] T 6fb9cad39c3844869e05df234c27cc9f P 4294638a7919452cb23c37e9f0a30fcb: No bootstrap required, opened a new log
I20260429 14:05:09.150590 14279 raft_consensus.cc:359] T 6fb9cad39c3844869e05df234c27cc9f P 5484942802e24c2291c633437870b3f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4294638a7919452cb23c37e9f0a30fcb" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 38095 } } peers { permanent_uuid: "5323c3aad1f64af6a392c1c1a59e3fa1" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 34339 } } peers { permanent_uuid: "5484942802e24c2291c633437870b3f6" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42027 } }
I20260429 14:05:09.150728 14279 raft_consensus.cc:385] T 6fb9cad39c3844869e05df234c27cc9f P 5484942802e24c2291c633437870b3f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:05:09.150766 14279 raft_consensus.cc:740] T 6fb9cad39c3844869e05df234c27cc9f P 5484942802e24c2291c633437870b3f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5484942802e24c2291c633437870b3f6, State: Initialized, Role: FOLLOWER
I20260429 14:05:09.150771 14280 ts_tablet_manager.cc:1403] T 6fb9cad39c3844869e05df234c27cc9f P 4294638a7919452cb23c37e9f0a30fcb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260429 14:05:09.150878 14279 consensus_queue.cc:260] T 6fb9cad39c3844869e05df234c27cc9f P 5484942802e24c2291c633437870b3f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4294638a7919452cb23c37e9f0a30fcb" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 38095 } } peers { permanent_uuid: "5323c3aad1f64af6a392c1c1a59e3fa1" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 34339 } } peers { permanent_uuid: "5484942802e24c2291c633437870b3f6" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42027 } }
I20260429 14:05:09.150942 14283 log.cc:826] T 6fb9cad39c3844869e05df234c27cc9f P 5323c3aad1f64af6a392c1c1a59e3fa1: Log is configured to *not* fsync() on all Append() calls
I20260429 14:05:09.151118 14279 ts_tablet_manager.cc:1434] T 6fb9cad39c3844869e05df234c27cc9f P 5484942802e24c2291c633437870b3f6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260429 14:05:09.151182 13988 heartbeater.cc:499] Master 127.2.197.254:35689 was elected leader, sending a full tablet report...
I20260429 14:05:09.151778 14283 tablet_bootstrap.cc:492] T 6fb9cad39c3844869e05df234c27cc9f P 5323c3aad1f64af6a392c1c1a59e3fa1: No bootstrap required, opened a new log
I20260429 14:05:09.151873 14283 ts_tablet_manager.cc:1403] T 6fb9cad39c3844869e05df234c27cc9f P 5323c3aad1f64af6a392c1c1a59e3fa1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260429 14:05:09.152995 14280 raft_consensus.cc:359] T 6fb9cad39c3844869e05df234c27cc9f P 4294638a7919452cb23c37e9f0a30fcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4294638a7919452cb23c37e9f0a30fcb" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 38095 } } peers { permanent_uuid: "5323c3aad1f64af6a392c1c1a59e3fa1" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 34339 } } peers { permanent_uuid: "5484942802e24c2291c633437870b3f6" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42027 } }
I20260429 14:05:09.153144 14280 raft_consensus.cc:385] T 6fb9cad39c3844869e05df234c27cc9f P 4294638a7919452cb23c37e9f0a30fcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:05:09.153175 14280 raft_consensus.cc:740] T 6fb9cad39c3844869e05df234c27cc9f P 4294638a7919452cb23c37e9f0a30fcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4294638a7919452cb23c37e9f0a30fcb, State: Initialized, Role: FOLLOWER
I20260429 14:05:09.153261 14280 consensus_queue.cc:260] T 6fb9cad39c3844869e05df234c27cc9f P 4294638a7919452cb23c37e9f0a30fcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4294638a7919452cb23c37e9f0a30fcb" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 38095 } } peers { permanent_uuid: "5323c3aad1f64af6a392c1c1a59e3fa1" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 34339 } } peers { permanent_uuid: "5484942802e24c2291c633437870b3f6" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42027 } }
I20260429 14:05:09.153550 14280 ts_tablet_manager.cc:1434] T 6fb9cad39c3844869e05df234c27cc9f P 4294638a7919452cb23c37e9f0a30fcb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260429 14:05:09.153627 14262 heartbeater.cc:499] Master 127.2.197.254:35689 was elected leader, sending a full tablet report...
I20260429 14:05:09.153998 14283 raft_consensus.cc:359] T 6fb9cad39c3844869e05df234c27cc9f P 5323c3aad1f64af6a392c1c1a59e3fa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4294638a7919452cb23c37e9f0a30fcb" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 38095 } } peers { permanent_uuid: "5323c3aad1f64af6a392c1c1a59e3fa1" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 34339 } } peers { permanent_uuid: "5484942802e24c2291c633437870b3f6" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42027 } }
I20260429 14:05:09.154103 14283 raft_consensus.cc:385] T 6fb9cad39c3844869e05df234c27cc9f P 5323c3aad1f64af6a392c1c1a59e3fa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:05:09.154121 14283 raft_consensus.cc:740] T 6fb9cad39c3844869e05df234c27cc9f P 5323c3aad1f64af6a392c1c1a59e3fa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5323c3aad1f64af6a392c1c1a59e3fa1, State: Initialized, Role: FOLLOWER
I20260429 14:05:09.154266 14283 consensus_queue.cc:260] T 6fb9cad39c3844869e05df234c27cc9f P 5323c3aad1f64af6a392c1c1a59e3fa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4294638a7919452cb23c37e9f0a30fcb" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 38095 } } peers { permanent_uuid: "5323c3aad1f64af6a392c1c1a59e3fa1" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 34339 } } peers { permanent_uuid: "5484942802e24c2291c633437870b3f6" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42027 } }
I20260429 14:05:09.154528 14283 ts_tablet_manager.cc:1434] T 6fb9cad39c3844869e05df234c27cc9f P 5323c3aad1f64af6a392c1c1a59e3fa1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260429 14:05:09.154570 14125 heartbeater.cc:499] Master 127.2.197.254:35689 was elected leader, sending a full tablet report...
W20260429 14:05:09.180410 14126 tablet.cc:2404] T 6fb9cad39c3844869e05df234c27cc9f P 5323c3aad1f64af6a392c1c1a59e3fa1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 14:05:09.215516 14284 raft_consensus.cc:493] T 6fb9cad39c3844869e05df234c27cc9f P 5484942802e24c2291c633437870b3f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:05:09.215641 14284 raft_consensus.cc:515] T 6fb9cad39c3844869e05df234c27cc9f P 5484942802e24c2291c633437870b3f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4294638a7919452cb23c37e9f0a30fcb" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 38095 } } peers { permanent_uuid: "5323c3aad1f64af6a392c1c1a59e3fa1" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 34339 } } peers { permanent_uuid: "5484942802e24c2291c633437870b3f6" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42027 } }
I20260429 14:05:09.215956 14284 leader_election.cc:290] T 6fb9cad39c3844869e05df234c27cc9f P 5484942802e24c2291c633437870b3f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4294638a7919452cb23c37e9f0a30fcb (127.2.197.195:38095), 5323c3aad1f64af6a392c1c1a59e3fa1 (127.2.197.194:34339)
I20260429 14:05:09.218999 13990 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:09.216121 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:38095 (local address 127.2.197.193:57871)
0429 14:05:09.216193 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0429 14:05:09.216215 (+ 22us) client_negotiation.cc:175] Beginning negotiation
0429 14:05:09.216285 (+ 70us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:05:09.216466 (+ 181us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:05:09.216467 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:05:09.216469 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:05:09.216747 (+ 278us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:09.216749 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:09.217760 (+ 1011us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:05:09.217760 (+ 0us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:05:09.218723 (+ 963us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:09.218726 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:09.218817 (+ 91us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:09.218822 (+ 5us) client_negotiation.cc:770] Sending connection context
0429 14:05:09.218859 (+ 37us) client_negotiation.cc:241] Negotiation successful
0429 14:05:09.218909 (+ 50us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260429 14:05:09.219008 14288 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:09.216247 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:34339 (local address 127.2.197.193:45795)
0429 14:05:09.216396 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0429 14:05:09.216406 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 14:05:09.216456 (+ 50us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:05:09.216614 (+ 158us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:05:09.216614 (+ 0us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:05:09.216616 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:05:09.216876 (+ 260us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:09.216877 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:09.217752 (+ 875us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:05:09.217754 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:05:09.218799 (+ 1045us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:05:09.218801 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:05:09.218865 (+ 64us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:09.218867 (+ 2us) client_negotiation.cc:770] Sending connection context
0429 14:05:09.218897 (+ 30us) client_negotiation.cc:241] Negotiation successful
0429 14:05:09.218922 (+ 25us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":75,"threads_started":1}
I20260429 14:05:09.219475 14278 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:09.216365 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:45795 (local address 127.2.197.194:34339)
0429 14:05:09.216439 (+ 74us) server_negotiation.cc:207] Beginning negotiation
0429 14:05:09.216441 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:05:09.216450 (+ 9us) server_negotiation.cc:408] Connection header received
0429 14:05:09.216527 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:05:09.216528 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:05:09.216537 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:05:09.216588 (+ 51us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:05:09.216957 (+ 369us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:09.217646 (+ 689us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:05:09.218884 (+ 1238us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:09.219349 (+ 465us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:09.219355 (+ 6us) server_negotiation.cc:1036] Waiting for connection context
0429 14:05:09.219379 (+ 24us) server_negotiation.cc:300] Negotiation successful
0429 14:05:09.219411 (+ 32us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260429 14:05:09.219475 14275 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:09.216238 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:57871 (local address 127.2.197.195:38095)
0429 14:05:09.216294 (+ 56us) server_negotiation.cc:207] Beginning negotiation
0429 14:05:09.216296 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:05:09.216308 (+ 12us) server_negotiation.cc:408] Connection header received
0429 14:05:09.216341 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:05:09.216342 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:05:09.216352 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:05:09.216447 (+ 95us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:05:09.216850 (+ 403us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:09.217622 (+ 772us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:05:09.218861 (+ 1239us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:09.219346 (+ 485us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:09.219355 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0429 14:05:09.219379 (+ 24us) server_negotiation.cc:300] Negotiation successful
0429 14:05:09.219412 (+ 33us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260429 14:05:09.219807 14216 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fb9cad39c3844869e05df234c27cc9f" candidate_uuid: "5484942802e24c2291c633437870b3f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4294638a7919452cb23c37e9f0a30fcb" is_pre_election: true
I20260429 14:05:09.219844 14079 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fb9cad39c3844869e05df234c27cc9f" candidate_uuid: "5484942802e24c2291c633437870b3f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5323c3aad1f64af6a392c1c1a59e3fa1" is_pre_election: true
I20260429 14:05:09.219972 14216 raft_consensus.cc:2468] T 6fb9cad39c3844869e05df234c27cc9f P 4294638a7919452cb23c37e9f0a30fcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5484942802e24c2291c633437870b3f6 in term 0.
I20260429 14:05:09.220001 14079 raft_consensus.cc:2468] T 6fb9cad39c3844869e05df234c27cc9f P 5323c3aad1f64af6a392c1c1a59e3fa1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5484942802e24c2291c633437870b3f6 in term 0.
I20260429 14:05:09.220219 13871 leader_election.cc:304] T 6fb9cad39c3844869e05df234c27cc9f P 5484942802e24c2291c633437870b3f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5323c3aad1f64af6a392c1c1a59e3fa1, 5484942802e24c2291c633437870b3f6; no voters:
I20260429 14:05:09.220353 14284 raft_consensus.cc:2804] T 6fb9cad39c3844869e05df234c27cc9f P 5484942802e24c2291c633437870b3f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 14:05:09.220407 14284 raft_consensus.cc:493] T 6fb9cad39c3844869e05df234c27cc9f P 5484942802e24c2291c633437870b3f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 14:05:09.220438 14284 raft_consensus.cc:3060] T 6fb9cad39c3844869e05df234c27cc9f P 5484942802e24c2291c633437870b3f6 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:05:09.221100 14284 raft_consensus.cc:515] T 6fb9cad39c3844869e05df234c27cc9f P 5484942802e24c2291c633437870b3f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4294638a7919452cb23c37e9f0a30fcb" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 38095 } } peers { permanent_uuid: "5323c3aad1f64af6a392c1c1a59e3fa1" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 34339 } } peers { permanent_uuid: "5484942802e24c2291c633437870b3f6" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42027 } }
I20260429 14:05:09.221251 14284 leader_election.cc:290] T 6fb9cad39c3844869e05df234c27cc9f P 5484942802e24c2291c633437870b3f6 [CANDIDATE]: Term 1 election: Requested vote from peers 4294638a7919452cb23c37e9f0a30fcb (127.2.197.195:38095), 5323c3aad1f64af6a392c1c1a59e3fa1 (127.2.197.194:34339)
I20260429 14:05:09.221455 14216 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fb9cad39c3844869e05df234c27cc9f" candidate_uuid: "5484942802e24c2291c633437870b3f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4294638a7919452cb23c37e9f0a30fcb"
I20260429 14:05:09.221450 14079 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fb9cad39c3844869e05df234c27cc9f" candidate_uuid: "5484942802e24c2291c633437870b3f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5323c3aad1f64af6a392c1c1a59e3fa1"
I20260429 14:05:09.221539 14216 raft_consensus.cc:3060] T 6fb9cad39c3844869e05df234c27cc9f P 4294638a7919452cb23c37e9f0a30fcb [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:05:09.221553 14079 raft_consensus.cc:3060] T 6fb9cad39c3844869e05df234c27cc9f P 5323c3aad1f64af6a392c1c1a59e3fa1 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:05:09.222432 14216 raft_consensus.cc:2468] T 6fb9cad39c3844869e05df234c27cc9f P 4294638a7919452cb23c37e9f0a30fcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5484942802e24c2291c633437870b3f6 in term 1.
I20260429 14:05:09.222432 14079 raft_consensus.cc:2468] T 6fb9cad39c3844869e05df234c27cc9f P 5323c3aad1f64af6a392c1c1a59e3fa1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5484942802e24c2291c633437870b3f6 in term 1.
I20260429 14:05:09.222667 13871 leader_election.cc:304] T 6fb9cad39c3844869e05df234c27cc9f P 5484942802e24c2291c633437870b3f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4294638a7919452cb23c37e9f0a30fcb, 5484942802e24c2291c633437870b3f6; no voters:
I20260429 14:05:09.222828 14284 raft_consensus.cc:2804] T 6fb9cad39c3844869e05df234c27cc9f P 5484942802e24c2291c633437870b3f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:05:09.223033 14284 raft_consensus.cc:697] T 6fb9cad39c3844869e05df234c27cc9f P 5484942802e24c2291c633437870b3f6 [term 1 LEADER]: Becoming Leader. State: Replica: 5484942802e24c2291c633437870b3f6, State: Running, Role: LEADER
I20260429 14:05:09.223172 14284 consensus_queue.cc:237] T 6fb9cad39c3844869e05df234c27cc9f P 5484942802e24c2291c633437870b3f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4294638a7919452cb23c37e9f0a30fcb" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 38095 } } peers { permanent_uuid: "5323c3aad1f64af6a392c1c1a59e3fa1" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 34339 } } peers { permanent_uuid: "5484942802e24c2291c633437870b3f6" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42027 } }
I20260429 14:05:09.224107 13790 catalog_manager.cc:5671] T 6fb9cad39c3844869e05df234c27cc9f P 5484942802e24c2291c633437870b3f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5484942802e24c2291c633437870b3f6 (127.2.197.193). New cstate: current_term: 1 leader_uuid: "5484942802e24c2291c633437870b3f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4294638a7919452cb23c37e9f0a30fcb" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 38095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5323c3aad1f64af6a392c1c1a59e3fa1" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5484942802e24c2291c633437870b3f6" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 42027 } health_report { overall_health: HEALTHY } } }
I20260429 14:05:09.238333 14276 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:09.235935 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35986 (local address 127.2.197.193:42027)
0429 14:05:09.236038 (+ 103us) server_negotiation.cc:207] Beginning negotiation
0429 14:05:09.236040 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:05:09.236049 (+ 9us) server_negotiation.cc:408] Connection header received
0429 14:05:09.236088 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:05:09.236089 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:05:09.236103 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:05:09.236163 (+ 60us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:05:09.236637 (+ 474us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:09.237114 (+ 477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:05:09.237749 (+ 635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:05:09.237884 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:05:09.237916 (+ 32us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:05:09.238003 (+ 87us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:05:09.238068 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0429 14:05:09.238194 (+ 126us) server_negotiation.cc:300] Negotiation successful
0429 14:05:09.238217 (+ 23us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260429 14:05:09.240585 14079 raft_consensus.cc:1275] T 6fb9cad39c3844869e05df234c27cc9f P 5323c3aad1f64af6a392c1c1a59e3fa1 [term 1 FOLLOWER]: Refusing update from remote peer 5484942802e24c2291c633437870b3f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:05:09.240615 14216 raft_consensus.cc:1275] T 6fb9cad39c3844869e05df234c27cc9f P 4294638a7919452cb23c37e9f0a30fcb [term 1 FOLLOWER]: Refusing update from remote peer 5484942802e24c2291c633437870b3f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:05:09.241050 14284 consensus_queue.cc:1048] T 6fb9cad39c3844869e05df234c27cc9f P 5484942802e24c2291c633437870b3f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5323c3aad1f64af6a392c1c1a59e3fa1" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:05:09.241192 14289 consensus_queue.cc:1048] T 6fb9cad39c3844869e05df234c27cc9f P 5484942802e24c2291c633437870b3f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4294638a7919452cb23c37e9f0a30fcb" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 38095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:05:09.244568 14294 mvcc.cc:204] Tried to move back new op lower bound from 7280523301846163456 to 7280523301778870272. Current Snapshot: MvccSnapshot[applied={T|T < 7280523301846163456}]
I20260429 14:05:09.246070 13790 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:05:09.246183 13790 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:05:09.247195 13790 catalog_manager.cc:5958] T 00000000000000000000000000000000 P b5504f4a0cfc430983fe26aa9478e57f: Sending DeleteTablet for 3 replicas of tablet 6fb9cad39c3844869e05df234c27cc9f
I20260429 14:05:09.247582 13921 tablet_service.cc:1558] Processing DeleteTablet for tablet 6fb9cad39c3844869e05df234c27cc9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:05:09 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:35976
I20260429 14:05:09.247701 14196 tablet_service.cc:1558] Processing DeleteTablet for tablet 6fb9cad39c3844869e05df234c27cc9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:05:09 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:51544
I20260429 14:05:09.247846 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 13854
I20260429 14:05:09.247920 14059 tablet_service.cc:1558] Processing DeleteTablet for tablet 6fb9cad39c3844869e05df234c27cc9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:05:09 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:47070
I20260429 14:05:09.248502 14304 tablet_replica.cc:333] T 6fb9cad39c3844869e05df234c27cc9f P 4294638a7919452cb23c37e9f0a30fcb: stopping tablet replica
I20260429 14:05:09.248608 14304 raft_consensus.cc:2243] T 6fb9cad39c3844869e05df234c27cc9f P 4294638a7919452cb23c37e9f0a30fcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:05:09.248632 14305 tablet_replica.cc:333] T 6fb9cad39c3844869e05df234c27cc9f P 5323c3aad1f64af6a392c1c1a59e3fa1: stopping tablet replica
I20260429 14:05:09.248723 14305 raft_consensus.cc:2243] T 6fb9cad39c3844869e05df234c27cc9f P 5323c3aad1f64af6a392c1c1a59e3fa1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:05:09.248775 14304 raft_consensus.cc:2272] T 6fb9cad39c3844869e05df234c27cc9f P 4294638a7919452cb23c37e9f0a30fcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:05:09.248845 14305 raft_consensus.cc:2272] T 6fb9cad39c3844869e05df234c27cc9f P 5323c3aad1f64af6a392c1c1a59e3fa1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:05:09.249341 14305 ts_tablet_manager.cc:1916] T 6fb9cad39c3844869e05df234c27cc9f P 5323c3aad1f64af6a392c1c1a59e3fa1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:05:09.249363 14304 ts_tablet_manager.cc:1916] T 6fb9cad39c3844869e05df234c27cc9f P 4294638a7919452cb23c37e9f0a30fcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:05:09.251241 14304 ts_tablet_manager.cc:1929] T 6fb9cad39c3844869e05df234c27cc9f P 4294638a7919452cb23c37e9f0a30fcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 14:05:09.251241 14305 ts_tablet_manager.cc:1929] T 6fb9cad39c3844869e05df234c27cc9f P 5323c3aad1f64af6a392c1c1a59e3fa1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 14:05:09.251322 14304 log.cc:1199] T 6fb9cad39c3844869e05df234c27cc9f P 4294638a7919452cb23c37e9f0a30fcb: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-2/wal/wals/6fb9cad39c3844869e05df234c27cc9f
I20260429 14:05:09.251322 14305 log.cc:1199] T 6fb9cad39c3844869e05df234c27cc9f P 5323c3aad1f64af6a392c1c1a59e3fa1: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777471413505459-2839-0/minicluster-data/ts-1/wal/wals/6fb9cad39c3844869e05df234c27cc9f
I20260429 14:05:09.251631 14304 ts_tablet_manager.cc:1950] T 6fb9cad39c3844869e05df234c27cc9f P 4294638a7919452cb23c37e9f0a30fcb: Deleting consensus metadata
I20260429 14:05:09.251717 14305 ts_tablet_manager.cc:1950] T 6fb9cad39c3844869e05df234c27cc9f P 5323c3aad1f64af6a392c1c1a59e3fa1: Deleting consensus metadata
I20260429 14:05:09.252130 13775 catalog_manager.cc:5002] TS 4294638a7919452cb23c37e9f0a30fcb (127.2.197.195:38095): tablet 6fb9cad39c3844869e05df234c27cc9f (table test-table [id=ed3317dfeb614a0893bb6c34e082f505]) successfully deleted
I20260429 14:05:09.252267 13775 catalog_manager.cc:5002] TS 5323c3aad1f64af6a392c1c1a59e3fa1 (127.2.197.194:34339): tablet 6fb9cad39c3844869e05df234c27cc9f (table test-table [id=ed3317dfeb614a0893bb6c34e082f505]) successfully deleted
W20260429 14:05:09.254163 13777 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.197.193:42027 (error 108)
I20260429 14:05:09.254354 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 13995
I20260429 14:05:09.255542 14274 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:05:09.255402 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:42027 (local address 127.0.0.1:35988)
0429 14:05:09.255468 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0429 14:05:09.255511 (+ 43us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.193:42027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260429 14:05:09.255663 13777 catalog_manager.cc:4729] TS 5484942802e24c2291c633437870b3f6 (127.2.197.193:42027): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6fb9cad39c3844869e05df234c27cc9f: Network error: Client connection negotiation failed: client connection to 127.2.197.193:42027: connect: Connection refused (error 111)
I20260429 14:05:09.260264 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 14132
I20260429 14:05:09.266726 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 13758
2026-04-29T14:05:09Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3095 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:05:09 dist-test-slave-8n77 krb5kdc[14309](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:05:09 dist-test-slave-8n77 krb5kdc[14309](info): set up 2 sockets
Apr 29 14:05:09 dist-test-slave-8n77 krb5kdc[14309](info): commencing operation
krb5kdc: starting...
W20260429 14:05:11.348590 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.059s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:05:11 dist-test-slave-8n77 krb5kdc[14309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:05:11Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:05:11Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.197.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.197.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.197.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777471413505459-2839-0/krb5kdc/rangeradmin_127.2.197.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.197.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777471413505459-2839-0/krb5kdc/rangeradmin_127.2.197.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.197.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.197.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.197.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777471413505459-2839-0/krb5kdc/rangerlookup_127.2.197.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.197.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777471413505459-2839-0/krb5kdc/rangerlookup_127.2.197.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.197.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777471413505459-2839-0/krb5kdc/HTTP_127.2.197.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.197.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777471413505459-2839-0/krb5kdc/HTTP_127.2.197.212@KRBTEST.COM.keytab.
I20260429 14:05:11.542223 2839 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777471413505459-2839-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskhpfkRA/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777471413505459-2839-0/minicluster-data/postgres -l logfile start
2026-04-29 14:05:14.746 UTC [14340] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2026-04-29 14:05:14.746 UTC [14340] LOG: listening on IPv4 address "127.2.197.212", port 45503
2026-04-29 14:05:14.751 UTC [14340] LOG: listening on Unix socket "/tmp/.s.PGSQL.45503"
2026-04-29 14:05:14.759 UTC [14345] LOG: database system was shut down at 2026-04-29 14:05:12 UTC
2026-04-29 14:05:14.765 UTC [14340] LOG: database system is ready to accept connections
I20260429 14:05:16.684006 2839 mini_postgres.cc:96] Postgres bound to 45503
2026-04-29 14:05:16.690 UTC [14351] FATAL: database "slave" does not exist
127.2.197.212:45503 - accepting connections
I20260429 14:05:16.690774 2839 mini_ranger.cc:161] Starting Ranger...
I20260429 14:05:16.710173 2839 mini_ranger.cc:84] Created miniranger Postgres user
I20260429 14:05:16.783633 2839 mini_ranger.cc:87] Created ranger Postgres database
I20260429 14:05:16.783741 2839 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777471413505459-2839-0/minicluster-data/ranger-admin
2026-04-29 14:05:17,086 [I] DB FLAVOR :POSTGRES
2026-04-29 14:05:17,087 [I] --------- Verifying Ranger DB connection ---------
2026-04-29 14:05:17,087 [I] Checking connection..
2026-04-29 14:05:17,087 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:45503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-04-29 14:05:17,417 [I] Checking connection passed.
2026-04-29 14:05:17,418 [I] --------- Verifying version history table ---------
2026-04-29 14:05:17,418 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:45503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-04-29 14:05:17,778 [I] Table x_db_version_h does not exist in database ranger
2026-04-29 14:05:17,778 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:45503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-04-29 14:05:18,123 [I] Table x_db_version_h does not exist in database ranger
2026-04-29 14:05:18,123 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-04-29 14:05:18,123 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:45503/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-04-29 14:05:18.445 UTC [14464] WARNING: there is no transaction in progress
2026-04-29 14:05:18,469 [I] create_dbversion_catalog.sql file imported successfully
2026-04-29 14:05:18,469 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:45503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-04-29 14:05:18,844 [I] Table x_db_version_h already exists in database 'ranger'
2026-04-29 14:05:18,845 [I] --------- Importing Ranger Core DB Schema ---------
2026-04-29 14:05:18,845 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:45503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-04-29 14:05:19,185 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:45503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-04-29 14:05:19,567 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:45503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8n77.c.gcp-upstream.internal','N') ;"
2026-04-29 14:05:19,894 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:45503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-04-29 14:05:20,235 [I] Table x_portal_user does not exist in database ranger
2026-04-29 14:05:20,235 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:45503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-04-29 14:05:20,555 [I] Table x_policy_ref_group does not exist in database ranger
2026-04-29 14:05:20,556 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:45503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-04-29 14:05:20,939 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:45503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-04-29 14:05:21,312 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-04-29 14:05:21,312 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:45503/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-04-29 14:05:22.308 UTC [14667] WARNING: there is no transaction in progress
2026-04-29 14:05:22.320 UTC [14667] WARNING: there is no transaction in progress
2026-04-29 14:05:22.334 UTC [14667] WARNING: there is no transaction in progress
2026-04-29 14:05:22.349 UTC [14667] WARNING: there is no transaction in progress
2026-04-29 14:05:22.366 UTC [14667] WARNING: there is no transaction in progress
2026-04-29 14:05:22.383 UTC [14667] WARNING: there is no transaction in progress
2026-04-29 14:05:22.451 UTC [14667] WARNING: there is no transaction in progress
2026-04-29 14:05:22.458 UTC [14667] WARNING: there is no transaction in progress
2026-04-29 14:05:22.471 UTC [14667] WARNING: there is no transaction in progress
2026-04-29 14:05:22.484 UTC [14667] WARNING: there is no transaction in progress
2026-04-29 14:05:22.497 UTC [14667] WARNING: there is no transaction in progress
2026-04-29 14:05:22.509 UTC [14667] WARNING: there is no transaction in progress
2026-04-29 14:05:22.519 UTC [14667] WARNING: there is no transaction in progress
2026-04-29 14:05:22.524 UTC [14667] WARNING: there is no transaction in progress
2026-04-29 14:05:22.531 UTC [14667] WARNING: there is no transaction in progress
2026-04-29 14:05:22.536 UTC [14667] WARNING: there is no transaction in progress
2026-04-29 14:05:23,155 [I] ranger_core_db_postgres.sql file imported successfully
2026-04-29 14:05:23,155 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:45503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-04-29 14:05:23,521 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-04-29 14:05:23,521 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:45503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-04-29 14:05:23,891 [I] Table x_portal_user already exists in database 'ranger'
2026-04-29 14:05:23,891 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:45503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-04-29 14:05:24,234 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-04-29 14:05:24,235 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:45503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-04-29 14:05:24,599 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:45503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-04-29 14:05:24,939 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:45503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8n77.c.gcp-upstream.internal';"
2026-04-29 14:05:25,277 [I] CORE_DB_SCHEMA import status has been updated
2026-04-29 14:05:25,278 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:45503/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-04-29 14:05:25,598 [I] DB_PATCHES have already been applied
I20260429 14:05:25.604516 2839 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777471413505459-2839-0/minicluster-data/ranger-admin:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/temurin-17-jdk-amd64/lib/*:/tmp/dist-test-taskhpfkRA/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260429 14:05:25.604597 2839 mini_ranger.cc:193] Using host: 127.2.197.212
I20260429 14:05:25.607615 2839 mini_ranger.cc:239] Ranger admin URL: http://127.2.197.212:50823
Apr 29, 2026 2:05:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Apr 29, 2026 2:05:26 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Apr 29, 2026 2:05:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777471413505459-2839-0/minicluster-data/ranger-admin/ews/webapp
Apr 29, 2026 2:05:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777471413505459-2839-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Apr 29, 2026 2:05:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777471413505459-2839-0/minicluster-data/ranger-admin/ews/webapp] .....
Apr 29, 2026 2:05:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777471413505459-2839-0/minicluster-data/ranger-admin/ews/webapp].
Apr 29, 2026 2:05:26 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
Apr 29, 2026 2:05:59 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Apr 29, 2026 2:05:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Apr 29, 2026 2:05:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Apr 29, 2026 2:05:59 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Apr 29, 2026 2:05:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Apr 29, 2026 2:05:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Apr 29, 2026 2:05:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Apr 29, 2026 2:05:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Apr 29, 2026 2:05:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Apr 29, 2026 2:05:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Apr 29, 2026 2:05:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Apr 29, 2026 2:05:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Apr 29, 2026 2:05:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Apr 29, 2026 2:05:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Apr 29, 2026 2:05:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Apr 29, 2026 2:05:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Apr 29, 2026 2:05:59 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Apr 29, 2026 2:06:00 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260429 14:07:27.138935 2839 mini_ranger.cc:160] Time spent starting Ranger: real 130.448s user 0.010s sys 0.084s
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/security-itest.cc:1052: Failure
Failed
Bad status: Timed out: Failed to start the Ranger service: process with PID 14826 is not yet bound to any port at the specified addresses; last attempt running lsof returned 'Not found: could not find pattern of a bound port in lsof output: p14826
f288
n127.0.0.1:43896->127.2.197.212:45503
f289
n127.0.0.1:43898->127.2.197.212:45503
f290
n127.0.0.1:43906->127.2.197.212:45503
f291
n127.0.0.1:43916->127.2.197.212:45503
f292
n127.0.0.1:43922->127.2.197.212:45503'
I20260429 14:07:27.140046 2839 mini_ranger.cc:66] Stopping Ranger...
I20260429 14:07:27.170811 2839 mini_ranger.cc:68] Stopped Ranger
2026-04-29 14:07:27.171 UTC [14340] LOG: received smart shutdown request
2026-04-29 14:07:27.174 UTC [14340] LOG: background worker "logical replication launcher" (PID 14348) exited with exit code 1
2026-04-29 14:07:27.175 UTC [14343] LOG: shutting down
2026-04-29 14:07:27.177 UTC [14343] LOG: checkpoint starting: shutdown immediate
2026-04-29 14:07:27.786 UTC [14343] LOG: checkpoint complete: wrote 1468 buffers (9.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.025 s, sync=0.578 s, total=0.612 s; sync files=815, longest=0.008 s, average=0.001 s; distance=8396 kB, estimate=8396 kB; lsn=0/1CADC28, redo lsn=0/1CADC28
2026-04-29 14:07:27.793 UTC [14340] LOG: database system is shut down
2026-04-29T14:07:27Z chronyd exiting
I20260429 14:07:27.811380 2839 test_util.cc:182] -----------------------------------------------
I20260429 14:07:27.811519 2839 test_util.cc:183] Had failures, leaving test files at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777471413505459-2839-0
[ FAILED ] SecurityITest.TestEncryptionWithKMSIntegration (138523 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:07:27 dist-test-slave-8n77 krb5kdc[14989](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:07:27 dist-test-slave-8n77 krb5kdc[14989](info): set up 2 sockets
Apr 29 14:07:27 dist-test-slave-8n77 krb5kdc[14989](info): commencing operation
krb5kdc: starting...
W20260429 14:07:29.823132 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:07:29 dist-test-slave-8n77 krb5kdc[14989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:07:29Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:07:29Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.197.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.197.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.197.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/rangeradmin_127.2.197.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.197.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/rangeradmin_127.2.197.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.197.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.197.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.197.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/rangerlookup_127.2.197.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.197.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/rangerlookup_127.2.197.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.197.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/HTTP_127.2.197.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.197.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/HTTP_127.2.197.212@KRBTEST.COM.keytab.
I20260429 14:07:30.001399 2839 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskhpfkRA/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/postgres -l logfile start
2026-04-29 14:07:33.056 UTC [15020] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2026-04-29 14:07:33.057 UTC [15020] LOG: listening on IPv4 address "127.2.197.212", port 38619
2026-04-29 14:07:33.063 UTC [15020] LOG: listening on Unix socket "/tmp/.s.PGSQL.38619"
2026-04-29 14:07:33.071 UTC [15025] LOG: database system was shut down at 2026-04-29 14:07:31 UTC
2026-04-29 14:07:33.078 UTC [15020] LOG: database system is ready to accept connections
I20260429 14:07:34.995097 2839 mini_postgres.cc:96] Postgres bound to 38619
2026-04-29 14:07:35.001 UTC [15031] FATAL: database "slave" does not exist
127.2.197.212:38619 - accepting connections
I20260429 14:07:35.001786 2839 mini_ranger.cc:161] Starting Ranger...
I20260429 14:07:35.020597 2839 mini_ranger.cc:84] Created miniranger Postgres user
I20260429 14:07:35.093712 2839 mini_ranger.cc:87] Created ranger Postgres database
I20260429 14:07:35.093808 2839 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ranger-admin
2026-04-29 14:07:35,397 [I] DB FLAVOR :POSTGRES
2026-04-29 14:07:35,398 [I] --------- Verifying Ranger DB connection ---------
2026-04-29 14:07:35,398 [I] Checking connection..
2026-04-29 14:07:35,398 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:38619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-04-29 14:07:35,714 [I] Checking connection passed.
2026-04-29 14:07:35,714 [I] --------- Verifying version history table ---------
2026-04-29 14:07:35,714 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:38619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-04-29 14:07:36,032 [I] Table x_db_version_h does not exist in database ranger
2026-04-29 14:07:36,033 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:38619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-04-29 14:07:36,357 [I] Table x_db_version_h does not exist in database ranger
2026-04-29 14:07:36,357 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-04-29 14:07:36,357 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:38619/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-04-29 14:07:36.641 UTC [15146] WARNING: there is no transaction in progress
2026-04-29 14:07:36,664 [I] create_dbversion_catalog.sql file imported successfully
2026-04-29 14:07:36,665 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:38619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-04-29 14:07:36,983 [I] Table x_db_version_h already exists in database 'ranger'
2026-04-29 14:07:36,983 [I] --------- Importing Ranger Core DB Schema ---------
2026-04-29 14:07:36,983 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:38619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-04-29 14:07:37,323 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:38619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-04-29 14:07:37,616 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:38619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8n77.c.gcp-upstream.internal','N') ;"
2026-04-29 14:07:37,976 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:38619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-04-29 14:07:38,285 [I] Table x_portal_user does not exist in database ranger
2026-04-29 14:07:38,286 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:38619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-04-29 14:07:38,617 [I] Table x_policy_ref_group does not exist in database ranger
2026-04-29 14:07:38,618 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:38619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-04-29 14:07:38,975 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:38619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-04-29 14:07:39,256 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-04-29 14:07:39,257 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:38619/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-04-29 14:07:40.211 UTC [15347] WARNING: there is no transaction in progress
2026-04-29 14:07:40.228 UTC [15347] WARNING: there is no transaction in progress
2026-04-29 14:07:40.246 UTC [15347] WARNING: there is no transaction in progress
2026-04-29 14:07:40.262 UTC [15347] WARNING: there is no transaction in progress
2026-04-29 14:07:40.277 UTC [15347] WARNING: there is no transaction in progress
2026-04-29 14:07:40.291 UTC [15347] WARNING: there is no transaction in progress
2026-04-29 14:07:40.346 UTC [15347] WARNING: there is no transaction in progress
2026-04-29 14:07:40.352 UTC [15347] WARNING: there is no transaction in progress
2026-04-29 14:07:40.363 UTC [15347] WARNING: there is no transaction in progress
2026-04-29 14:07:40.374 UTC [15347] WARNING: there is no transaction in progress
2026-04-29 14:07:40.385 UTC [15347] WARNING: there is no transaction in progress
2026-04-29 14:07:40.396 UTC [15347] WARNING: there is no transaction in progress
2026-04-29 14:07:40.406 UTC [15347] WARNING: there is no transaction in progress
2026-04-29 14:07:40.411 UTC [15347] WARNING: there is no transaction in progress
2026-04-29 14:07:40.416 UTC [15347] WARNING: there is no transaction in progress
2026-04-29 14:07:40.421 UTC [15347] WARNING: there is no transaction in progress
2026-04-29 14:07:40,960 [I] ranger_core_db_postgres.sql file imported successfully
2026-04-29 14:07:40,960 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:38619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-04-29 14:07:41,284 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-04-29 14:07:41,284 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:38619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-04-29 14:07:41,599 [I] Table x_portal_user already exists in database 'ranger'
2026-04-29 14:07:41,599 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:38619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-04-29 14:07:41,958 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-04-29 14:07:41,959 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:38619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-04-29 14:07:42,294 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:38619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-04-29 14:07:42,666 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:38619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8n77.c.gcp-upstream.internal';"
2026-04-29 14:07:43,006 [I] CORE_DB_SCHEMA import status has been updated
2026-04-29 14:07:43,007 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:38619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-04-29 14:07:43,325 [I] DB_PATCHES have already been applied
I20260429 14:07:43.331202 2839 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ranger-admin:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/temurin-17-jdk-amd64/lib/*:/tmp/dist-test-taskhpfkRA/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260429 14:07:43.331278 2839 mini_ranger.cc:193] Using host: 127.2.197.212
I20260429 14:07:43.333999 2839 mini_ranger.cc:239] Ranger admin URL: http://127.2.197.212:33825
Apr 29, 2026 2:07:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Apr 29, 2026 2:07:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Apr 29, 2026 2:07:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ranger-admin/ews/webapp
Apr 29, 2026 2:07:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Apr 29, 2026 2:07:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ranger-admin/ews/webapp] .....
Apr 29, 2026 2:07:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ranger-admin/ews/webapp].
Apr 29, 2026 2:07:44 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260429 14:07:45.637701 2839 mini_ranger.cc:160] Time spent starting Ranger: real 10.636s user 0.000s sys 0.010s
Apr 29, 2026 2:08:10 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Apr 29, 2026 2:08:10 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.UserREST
Apr 29, 2026 2:08:10 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Apr 29, 2026 2:08:11 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Apr 29, 2026 2:08:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Apr 29, 2026 2:08:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Apr 29, 2026 2:08:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Apr 29, 2026 2:08:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Apr 29, 2026 2:08:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Apr 29, 2026 2:08:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Apr 29, 2026 2:08:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Apr 29, 2026 2:08:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Apr 29, 2026 2:08:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Apr 29, 2026 2:08:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Apr 29, 2026 2:08:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Apr 29, 2026 2:08:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Apr 29, 2026 2:08:11 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Apr 29, 2026 2:08:12 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260429 14:08:12.723737 2839 mini_ranger.cc:273] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.197.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.197.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.197.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/rangerkms_127.2.197.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.197.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/rangerkms_127.2.197.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.197.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.197.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/HTTP_127.2.197.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.197.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/HTTP_127.2.197.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Apr 29 14:08:12 dist-test-slave-8n77 krb5kdc[14989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471692, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260429 14:08:12.832903 2839 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260429 14:08:12.845642 2839 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260429 14:08:12.978787 2839 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260429 14:08:12.978869 2839 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ranger-kms
I20260429 14:08:12.978899 2839 mini_ranger_kms.cc:227] Using postgres at 127.2.197.212:38619
2026-04-29 14:08:13,212 [I] DB FLAVOR :POSTGRES
2026-04-29 14:08:13,212 [I] --------- Verifying Ranger DB connection ---------
2026-04-29 14:08:13,212 [I] Checking connection
2026-04-29 14:08:13,213 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:38619/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-04-29 14:08:13,640 [I] connection success
2026-04-29 14:08:13,640 [I] --------- Verifying Ranger DB tables ---------
2026-04-29 14:08:13,640 [I] Verifying table ranger_masterkey in database rangerkms
2026-04-29 14:08:13,640 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:38619/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-04-29 14:08:13,990 [I] Table ranger_masterkey does not exist in database rangerkms
2026-04-29 14:08:13,990 [I] --------- Importing Ranger Core DB Schema ---------
2026-04-29 14:08:13,990 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-04-29 14:08:13,991 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.197.212:38619/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-04-29 14:08:14,349 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260429 14:08:14.915930 2839 mini_ranger_kms.cc:326] Created kms service
I20260429 14:08:15.151959 2839 mini_ranger_kms.cc:342] Created kudu user
I20260429 14:08:15.209380 2839 mini_ranger_kms.cc:359] Created rangerkms user
Apr 29 14:08:15 dist-test-slave-8n77 krb5kdc[14989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471692, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.197.212@KRBTEST.COM
I20260429 14:08:15.708356 2839 mini_ranger_kms.cc:400] Added ranger policy
I20260429 14:08:15.708520 2839 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ranger-kms:/tmp/dist-test-taskhpfkRA/build/release/bin/postgresql.jar:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskhpfkRA/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/temurin-17-jdk-amd64/lib/*
I20260429 14:08:15.708544 2839 mini_ranger_kms.cc:240] Using host: 127.2.197.212
I20260429 14:08:15.713492 2839 mini_ranger_kms.cc:292] Ranger KMS PID: 15651
I20260429 14:08:15.713598 2839 mini_ranger_kms.cc:293] Ranger KMS URL: http://127.2.197.212:39899
Apr 29, 2026 2:08:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Apr 29, 2026 2:08:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Apr 29, 2026 2:08:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Apr 29, 2026 2:08:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ranger-kms/ews/webapp] .....
Apr 29, 2026 2:08:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ranger-kms/ews/webapp].
Apr 29, 2026 2:08:17 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260429 14:08:18.051028 2839 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.218s user 0.007s sys 0.190s
I20260429 14:08:18.051141 2839 mini_ranger_kms.cc:413] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260429 14:08:18.051192 2839 mini_ranger_kms.cc:417] 127.2.197.212:39899/kms/v1/keys
Apr 29 14:08:18 dist-test-slave-8n77 krb5kdc[14989](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777471698, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.197.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Apr 29 14:08:19 dist-test-slave-8n77 krb5kdc[14989](info): TGS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777471698, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.197.212@KRBTEST.COM for HTTP/127.2.197.212@KRBTEST.COM
Apr 29 14:08:19 dist-test-slave-8n77 krb5kdc[14989](info): TGS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777471698, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.197.212@KRBTEST.COM for HTTP/127.2.197.212@KRBTEST.COM
Apr 29 14:08:22 dist-test-slave-8n77 krb5kdc[14989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:08:22.170738 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:40767
--webserver_interface=127.2.197.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:40767
--ranger_config_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.197.212:39899/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:08:22.259229 15719 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:08:22.259476 15719 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:08:22.259518 15719 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:08:22.261323 15719 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:08:22.261461 15719 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:08:22.261492 15719 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:08:22.261540 15719 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:08:22.261583 15719 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:08:22.263741 15719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.197.212:39899/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:40767
--ranger_config_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:40767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:08:22.264055 15719 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:08:22.264261 15719 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:08:22.267549 15724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:22.267699 15727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:22.267602 15725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:08:22.267983 15719 server_base.cc:1061] running on GCE node
I20260429 14:08:22.268235 15719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:08:22.268501 15719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:08:22.269708 15719 hybrid_clock.cc:648] HybridClock initialized: now 1777471702269691 us; error 29 us; skew 500 ppm
Apr 29 14:08:22 dist-test-slave-8n77 krb5kdc[14989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471702, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:08:22.272374 15719 init.cc:377] Logged in from keytab as kudu/127.2.197.254@KRBTEST.COM (short username kudu)
I20260429 14:08:22.273025 15719 webserver.cc:492] Webserver started at http://127.2.197.254:41761/ using document root <none> and password file <none>
I20260429 14:08:22.273239 15719 fs_manager.cc:362] Metadata directory not provided
I20260429 14:08:22.273300 15719 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:08:22.273411 15719 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
Apr 29 14:08:22 dist-test-slave-8n77 krb5kdc[14989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471702, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for HTTP/127.2.197.212@KRBTEST.COM
I20260429 14:08:22.325261 15719 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "b9020c4422be4e73988c6bff8803a13e"
format_stamp: "Formatted at 2026-04-29 14:08:22 on dist-test-slave-8n77"
server_key: "67dbf21c9b9998539831bef285f0ca5c"
server_key_iv: "33976cf9ea813b02f96abb708f5cafdc"
server_key_version: "kuduclusterkey@0"
I20260429 14:08:22.325788 15719 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "b9020c4422be4e73988c6bff8803a13e"
format_stamp: "Formatted at 2026-04-29 14:08:22 on dist-test-slave-8n77"
server_key: "67dbf21c9b9998539831bef285f0ca5c"
server_key_iv: "33976cf9ea813b02f96abb708f5cafdc"
server_key_version: "kuduclusterkey@0"
I20260429 14:08:22.328048 15719 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260429 14:08:22.342644 15734 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:22.342878 15719 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:08:22.342958 15719 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "b9020c4422be4e73988c6bff8803a13e"
format_stamp: "Formatted at 2026-04-29 14:08:22 on dist-test-slave-8n77"
server_key: "67dbf21c9b9998539831bef285f0ca5c"
server_key_iv: "33976cf9ea813b02f96abb708f5cafdc"
server_key_version: "kuduclusterkey@0"
I20260429 14:08:22.343060 15719 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:08:22.353271 15719 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:08:22.358561 15719 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:08:22.358743 15719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:08:22.363997 15719 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:40767
I20260429 14:08:22.364014 15787 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:40767 every 8 connection(s)
I20260429 14:08:22.364391 15719 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:08:22.365061 15788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:08:22.366670 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 15719
I20260429 14:08:22.366811 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:08:22.366998 2839 external_mini_cluster.cc:1468] Setting key 4df1d836b1b3b279b21b94d8afdae076
I20260429 14:08:22.367864 15788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e: Bootstrap starting.
I20260429 14:08:22.368530 15788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e: Neither blocks nor log segments found. Creating new log.
I20260429 14:08:22.368788 15788 log.cc:826] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e: Log is configured to *not* fsync() on all Append() calls
I20260429 14:08:22.369628 15788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e: No bootstrap required, opened a new log
I20260429 14:08:22.371639 15788 raft_consensus.cc:359] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9020c4422be4e73988c6bff8803a13e" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40767 } }
I20260429 14:08:22.371835 15788 raft_consensus.cc:385] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:08:22.371886 15788 raft_consensus.cc:740] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9020c4422be4e73988c6bff8803a13e, State: Initialized, Role: FOLLOWER
I20260429 14:08:22.372010 15788 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9020c4422be4e73988c6bff8803a13e" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40767 } }
I20260429 14:08:22.372074 15788 raft_consensus.cc:399] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:08:22.372102 15788 raft_consensus.cc:493] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:08:22.372136 15788 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:08:22.372849 15788 raft_consensus.cc:515] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9020c4422be4e73988c6bff8803a13e" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40767 } }
I20260429 14:08:22.373001 15788 leader_election.cc:304] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9020c4422be4e73988c6bff8803a13e; no voters:
I20260429 14:08:22.373168 15788 leader_election.cc:290] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:08:22.373219 15793 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:08:22.373474 15793 raft_consensus.cc:697] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e [term 1 LEADER]: Becoming Leader. State: Replica: b9020c4422be4e73988c6bff8803a13e, State: Running, Role: LEADER
Apr 29 14:08:22 dist-test-slave-8n77 krb5kdc[14989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:08:22.373696 15793 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9020c4422be4e73988c6bff8803a13e" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40767 } }
I20260429 14:08:22.373473 15788 sys_catalog.cc:565] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:08:22.374068 15794 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9020c4422be4e73988c6bff8803a13e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9020c4422be4e73988c6bff8803a13e" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40767 } } }
I20260429 14:08:22.374137 15794 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e [sys.catalog]: This master's current role is: LEADER
I20260429 14:08:22.374374 15795 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9020c4422be4e73988c6bff8803a13e. Latest consensus state: current_term: 1 leader_uuid: "b9020c4422be4e73988c6bff8803a13e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9020c4422be4e73988c6bff8803a13e" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40767 } } }
I20260429 14:08:22.374486 15795 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e [sys.catalog]: This master's current role is: LEADER
I20260429 14:08:22.374620 15797 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:08:22.375077 15797 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:08:22.375752 15788 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260429 14:08:22.381796 15791 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:22.367595 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:39732 (local address 127.2.197.254:40767)
0429 14:08:22.367757 (+ 162us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:22.367771 (+ 14us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:22.367797 (+ 26us) server_negotiation.cc:408] Connection header received
0429 14:08:22.368167 (+ 370us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:22.368174 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:22.368215 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:22.368357 (+ 142us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:22.369481 (+ 1124us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:22.370830 (+ 1349us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:22.371449 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:22.371734 (+ 285us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:22.374160 (+ 2426us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:22.374168 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:22.374169 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:22.374181 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:22.378375 (+ 4194us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:22.378950 (+ 575us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:22.378957 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:22.378957 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:22.379063 (+ 106us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:22.379269 (+ 206us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:22.379270 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:22.379271 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:22.379786 (+ 515us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:22.380075 (+ 289us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:22.380711 (+ 636us) server_negotiation.cc:300] Negotiation successful
0429 14:08:22.381090 (+ 379us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":67,"threads_started":1}
I20260429 14:08:22.383335 15797 catalog_manager.cc:1357] Generated new cluster ID: 636bd676e72e47c695cdb2e901cff063
I20260429 14:08:22.383432 15797 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:08:22.400560 15797 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:08:22.401419 15797 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:08:22.407347 15797 catalog_manager.cc:6044] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e: Generated new TSK 0
I20260429 14:08:22.407639 15797 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:08:23.269593 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.197.212:39899/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.193:0
--local_ip_for_outbound_sockets=127.2.197.193
--webserver_interface=127.2.197.193
--webserver_port=0
--tserver_master_addrs=127.2.197.254:40767
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:08:23.365804 15847 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:08:23.366031 15847 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:08:23.366073 15847 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:08:23.368695 15847 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:08:23.368829 15847 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:08:23.368877 15847 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.193
I20260429 14:08:23.371166 15847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.197.212:39899/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.197.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:40767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.193
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:08:23.371592 15847 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:08:23.371833 15847 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:08:23.375093 15853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:23.375111 15852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:23.375430 15855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:08:23.375682 15847 server_base.cc:1061] running on GCE node
I20260429 14:08:23.375885 15847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:08:23.376133 15847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:08:23.377322 15847 hybrid_clock.cc:648] HybridClock initialized: now 1777471703377287 us; error 48 us; skew 500 ppm
Apr 29 14:08:23 dist-test-slave-8n77 krb5kdc[14989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471703, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:08:23.380095 15847 init.cc:377] Logged in from keytab as kudu/127.2.197.193@KRBTEST.COM (short username kudu)
I20260429 14:08:23.380704 15847 webserver.cc:492] Webserver started at http://127.2.197.193:36745/ using document root <none> and password file <none>
I20260429 14:08:23.380925 15847 fs_manager.cc:362] Metadata directory not provided
I20260429 14:08:23.380987 15847 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:08:23.381127 15847 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
Apr 29 14:08:23 dist-test-slave-8n77 krb5kdc[14989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471703, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for HTTP/127.2.197.212@KRBTEST.COM
I20260429 14:08:23.407730 15847 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-0/data/instance:
uuid: "5631cd0a3f9448c2889a1b0872328418"
format_stamp: "Formatted at 2026-04-29 14:08:23 on dist-test-slave-8n77"
server_key: "505902956ed3d37899f7bed8cc982e8a"
server_key_iv: "f4333992a0b455918720a4c983a1cf08"
server_key_version: "kuduclusterkey@0"
I20260429 14:08:23.408188 15847 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance:
uuid: "5631cd0a3f9448c2889a1b0872328418"
format_stamp: "Formatted at 2026-04-29 14:08:23 on dist-test-slave-8n77"
server_key: "505902956ed3d37899f7bed8cc982e8a"
server_key_iv: "f4333992a0b455918720a4c983a1cf08"
server_key_version: "kuduclusterkey@0"
I20260429 14:08:23.413268 15847 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260429 14:08:23.442489 15864 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:23.442813 15847 fs_manager.cc:730] Time spent opening block manager: real 0.012s user 0.001s sys 0.000s
I20260429 14:08:23.442895 15847 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-0/wal
uuid: "5631cd0a3f9448c2889a1b0872328418"
format_stamp: "Formatted at 2026-04-29 14:08:23 on dist-test-slave-8n77"
server_key: "505902956ed3d37899f7bed8cc982e8a"
server_key_iv: "f4333992a0b455918720a4c983a1cf08"
server_key_version: "kuduclusterkey@0"
I20260429 14:08:23.442981 15847 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:08:23.494856 15847 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:08:23.498749 15847 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:08:23.498978 15847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:08:23.499212 15847 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:08:23.499655 15847 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:08:23.499756 15847 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:23.499862 15847 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:08:23.499938 15847 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:23.508127 15847 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.193:41755
I20260429 14:08:23.508152 15979 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.193:41755 every 8 connection(s)
I20260429 14:08:23.508585 15847 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
Apr 29 14:08:23 dist-test-slave-8n77 krb5kdc[14989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471703, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:08:23.516619 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 15847
I20260429 14:08:23.516741 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance
I20260429 14:08:23.516914 2839 external_mini_cluster.cc:1468] Setting key 7a7328bf44f9f952b3dd94f2e6b204a0
I20260429 14:08:23.523729 15982 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:23.509392 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:38983 (local address 127.2.197.254:40767)
0429 14:08:23.511153 (+ 1761us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:23.511155 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:23.511163 (+ 8us) server_negotiation.cc:408] Connection header received
0429 14:08:23.511189 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:23.511190 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:23.511202 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:23.512468 (+ 1266us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:23.512513 (+ 45us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:23.513027 (+ 514us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:23.513620 (+ 593us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:23.513816 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:23.515745 (+ 1929us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:23.515750 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:23.515750 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:23.515760 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:23.518110 (+ 2350us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:23.522548 (+ 4438us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:23.522550 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:23.522550 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:23.522592 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:23.522873 (+ 281us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:23.522874 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:23.522874 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:23.523008 (+ 134us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:23.523070 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:23.523460 (+ 390us) server_negotiation.cc:300] Negotiation successful
0429 14:08:23.523584 (+ 124us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1735,"thread_start_us":65,"threads_started":1}
I20260429 14:08:23.524380 15983 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:23.509389 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40767 (local address 127.2.197.193:38983)
0429 14:08:23.509647 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:23.509681 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:23.510851 (+ 1170us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:23.511355 (+ 504us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:23.511361 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:23.511597 (+ 236us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:08:23.512331 (+ 734us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:23.512334 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:23.513130 (+ 796us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:23.513132 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:23.513525 (+ 393us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:23.513527 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:23.513685 (+ 158us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:23.514321 (+ 636us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:08:23.514334 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:08:23.515650 (+ 1316us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:08:23.518214 (+ 2564us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:23.518228 (+ 14us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:23.518230 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:23.522371 (+ 4141us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:23.522671 (+ 300us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:23.522673 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:23.522673 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:23.522788 (+ 115us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:23.523107 (+ 319us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:08:23.523113 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:08:23.523283 (+ 170us) client_negotiation.cc:770] Sending connection context
0429 14:08:23.523893 (+ 610us) client_negotiation.cc:241] Negotiation successful
0429 14:08:23.524094 (+ 201us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":74,"threads_started":1}
I20260429 14:08:23.524977 15980 heartbeater.cc:344] Connected to a master server at 127.2.197.254:40767
I20260429 14:08:23.525098 15980 heartbeater.cc:461] Registering TS with master...
I20260429 14:08:23.525309 15980 heartbeater.cc:507] Master 127.2.197.254:40767 requested a full tablet report, sending...
I20260429 14:08:23.525844 15751 ts_manager.cc:194] Registered new tserver with Master: 5631cd0a3f9448c2889a1b0872328418 (127.2.197.193:41755)
I20260429 14:08:23.526861 15751 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.193@KRBTEST.COM'} at 127.2.197.193:38983
WARNING: no policy specified for kudu/127.2.197.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:08:23.614950 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.197.212:39899/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.194:0
--local_ip_for_outbound_sockets=127.2.197.194
--webserver_interface=127.2.197.194
--webserver_port=0
--tserver_master_addrs=127.2.197.254:40767
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Apr 29 14:08:23 dist-test-slave-8n77 krb5kdc[14989](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777471703, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260429 14:08:23.737919 15990 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:08:23.738256 15990 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:08:23.738350 15990 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:08:23.740113 15990 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:08:23.740208 15990 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:08:23.740254 15990 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.194
I20260429 14:08:23.742228 15990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.197.212:39899/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.197.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:40767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.194
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:08:23.742635 15990 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:08:23.742856 15990 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:08:23.745596 15996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:23.745770 15995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:23.746852 15998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:08:23.747493 15990 server_base.cc:1061] running on GCE node
I20260429 14:08:23.747735 15990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:08:23.747970 15990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:08:23.749136 15990 hybrid_clock.cc:648] HybridClock initialized: now 1777471703749112 us; error 40 us; skew 500 ppm
Apr 29 14:08:23 dist-test-slave-8n77 krb5kdc[14989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471703, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:08:23.751961 15990 init.cc:377] Logged in from keytab as kudu/127.2.197.194@KRBTEST.COM (short username kudu)
I20260429 14:08:23.752751 15990 webserver.cc:492] Webserver started at http://127.2.197.194:34635/ using document root <none> and password file <none>
I20260429 14:08:23.752998 15990 fs_manager.cc:362] Metadata directory not provided
I20260429 14:08:23.753060 15990 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:08:23.753167 15990 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
Apr 29 14:08:23 dist-test-slave-8n77 krb5kdc[14989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471703, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for HTTP/127.2.197.212@KRBTEST.COM
I20260429 14:08:23.768631 15990 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-1/data/instance:
uuid: "8bfed081c9264b66908c0cc446b51bf8"
format_stamp: "Formatted at 2026-04-29 14:08:23 on dist-test-slave-8n77"
server_key: "79daa690ad8b1dba5a1556e602a06424"
server_key_iv: "70a749d6c686f2825c714becd14c3770"
server_key_version: "kuduclusterkey@0"
I20260429 14:08:23.769330 15990 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance:
uuid: "8bfed081c9264b66908c0cc446b51bf8"
format_stamp: "Formatted at 2026-04-29 14:08:23 on dist-test-slave-8n77"
server_key: "79daa690ad8b1dba5a1556e602a06424"
server_key_iv: "70a749d6c686f2825c714becd14c3770"
server_key_version: "kuduclusterkey@0"
I20260429 14:08:23.771364 15990 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260429 14:08:23.811328 16006 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:23.811654 15990 fs_manager.cc:730] Time spent opening block manager: real 0.013s user 0.000s sys 0.000s
I20260429 14:08:23.811739 15990 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-1/wal
uuid: "8bfed081c9264b66908c0cc446b51bf8"
format_stamp: "Formatted at 2026-04-29 14:08:23 on dist-test-slave-8n77"
server_key: "79daa690ad8b1dba5a1556e602a06424"
server_key_iv: "70a749d6c686f2825c714becd14c3770"
server_key_version: "kuduclusterkey@0"
I20260429 14:08:23.811829 15990 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:08:23.837337 15990 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:08:23.850230 15990 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:08:23.850508 15990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:08:23.850791 15990 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:08:23.851210 15990 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:08:23.851306 15990 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:23.851382 15990 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:08:23.851441 15990 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:23.864713 15990 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.194:43035
I20260429 14:08:23.865231 15990 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
I20260429 14:08:23.872200 16121 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.194:43035 every 8 connection(s)
I20260429 14:08:23.879278 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 15990
I20260429 14:08:23.879418 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance
I20260429 14:08:23.879591 2839 external_mini_cluster.cc:1468] Setting key 53f08cba87a13790703f7ccc288a4e0e
Apr 29 14:08:23 dist-test-slave-8n77 krb5kdc[14989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471703, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:08:23.895888 15982 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:23.873369 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:46401 (local address 127.2.197.254:40767)
0429 14:08:23.873416 (+ 47us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:23.873419 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:23.878607 (+ 5188us) server_negotiation.cc:408] Connection header received
0429 14:08:23.879946 (+ 1339us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:23.879948 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:23.879966 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:23.879998 (+ 32us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:23.881071 (+ 1073us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:23.881918 (+ 847us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:23.887276 (+ 5358us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:23.887455 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:23.892115 (+ 4660us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:23.892122 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:23.892123 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:23.892133 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:23.894020 (+ 1887us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:23.894668 (+ 648us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:23.894669 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:23.894670 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:23.894699 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:23.894933 (+ 234us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:23.894934 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:23.894935 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:23.895055 (+ 120us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:23.895110 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:23.895534 (+ 424us) server_negotiation.cc:300] Negotiation successful
0429 14:08:23.895772 (+ 238us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260429 14:08:23.896771 16124 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:23.873070 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40767 (local address 127.2.197.194:46401)
0429 14:08:23.878427 (+ 5357us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:23.878485 (+ 58us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:23.879805 (+ 1320us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:23.880196 (+ 391us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:23.880201 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:23.880508 (+ 307us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:08:23.880948 (+ 440us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:23.880953 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:23.886461 (+ 5508us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:23.886463 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:23.887132 (+ 669us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:23.887135 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:23.887371 (+ 236us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:23.889442 (+ 2071us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:08:23.889454 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:08:23.891954 (+ 2500us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:08:23.894184 (+ 2230us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:23.894194 (+ 10us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:23.894197 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:23.894553 (+ 356us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:23.894769 (+ 216us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:23.894769 (+ 0us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:23.894770 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:23.894859 (+ 89us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:23.895177 (+ 318us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:08:23.895183 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:08:23.895357 (+ 174us) client_negotiation.cc:770] Sending connection context
0429 14:08:23.896289 (+ 932us) client_negotiation.cc:241] Negotiation successful
0429 14:08:23.896491 (+ 202us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5295,"thread_start_us":112,"threads_started":1}
I20260429 14:08:23.897486 16122 heartbeater.cc:344] Connected to a master server at 127.2.197.254:40767
I20260429 14:08:23.897588 16122 heartbeater.cc:461] Registering TS with master...
I20260429 14:08:23.897814 16122 heartbeater.cc:507] Master 127.2.197.254:40767 requested a full tablet report, sending...
I20260429 14:08:23.898267 15751 ts_manager.cc:194] Registered new tserver with Master: 8bfed081c9264b66908c0cc446b51bf8 (127.2.197.194:43035)
I20260429 14:08:23.898844 15751 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.194@KRBTEST.COM'} at 127.2.197.194:46401
WARNING: no policy specified for kudu/127.2.197.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:08:23.981202 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.197.212:39899/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.195:0
--local_ip_for_outbound_sockets=127.2.197.195
--webserver_interface=127.2.197.195
--webserver_port=0
--tserver_master_addrs=127.2.197.254:40767
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:08:24.098551 16129 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:08:24.098894 16129 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:08:24.098975 16129 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:08:24.101409 16129 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:08:24.101557 16129 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:08:24.101657 16129 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.195
I20260429 14:08:24.104393 16129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.197.212:39899/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.197.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:40767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.195
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:08:24.104895 16129 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:08:24.105202 16129 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:08:24.108367 16137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:24.108737 16135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:24.109287 16134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:08:24.110229 16129 server_base.cc:1061] running on GCE node
I20260429 14:08:24.110543 16129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:08:24.110795 16129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:08:24.111931 16129 hybrid_clock.cc:648] HybridClock initialized: now 1777471704111899 us; error 43 us; skew 500 ppm
Apr 29 14:08:24 dist-test-slave-8n77 krb5kdc[14989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471704, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:08:24.116268 16129 init.cc:377] Logged in from keytab as kudu/127.2.197.195@KRBTEST.COM (short username kudu)
I20260429 14:08:24.117062 16129 webserver.cc:492] Webserver started at http://127.2.197.195:40979/ using document root <none> and password file <none>
I20260429 14:08:24.117332 16129 fs_manager.cc:362] Metadata directory not provided
I20260429 14:08:24.117439 16129 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:08:24.117586 16129 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
Apr 29 14:08:24 dist-test-slave-8n77 krb5kdc[14989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471704, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for HTTP/127.2.197.212@KRBTEST.COM
I20260429 14:08:24.137914 16129 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-2/data/instance:
uuid: "039b92e0728d49828d1f6cad35a809e4"
format_stamp: "Formatted at 2026-04-29 14:08:24 on dist-test-slave-8n77"
server_key: "55732e10623447bbc613f4fa507a3a49"
server_key_iv: "b2f4f47ab8b8d4325b33c5f471d35623"
server_key_version: "kuduclusterkey@0"
I20260429 14:08:24.138558 16129 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance:
uuid: "039b92e0728d49828d1f6cad35a809e4"
format_stamp: "Formatted at 2026-04-29 14:08:24 on dist-test-slave-8n77"
server_key: "55732e10623447bbc613f4fa507a3a49"
server_key_iv: "b2f4f47ab8b8d4325b33c5f471d35623"
server_key_version: "kuduclusterkey@0"
I20260429 14:08:24.142719 16129 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.002s
I20260429 14:08:24.156697 16145 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:24.156961 16129 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:08:24.157030 16129 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-2/wal
uuid: "039b92e0728d49828d1f6cad35a809e4"
format_stamp: "Formatted at 2026-04-29 14:08:24 on dist-test-slave-8n77"
server_key: "55732e10623447bbc613f4fa507a3a49"
server_key_iv: "b2f4f47ab8b8d4325b33c5f471d35623"
server_key_version: "kuduclusterkey@0"
I20260429 14:08:24.157109 16129 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:08:24.197652 16129 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:08:24.207809 16129 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:08:24.207953 16129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:08:24.208119 16129 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:08:24.208443 16129 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:08:24.208479 16129 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:24.208510 16129 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:08:24.208528 16129 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:24.216765 16129 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.195:39745
I20260429 14:08:24.217290 16129 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
I20260429 14:08:24.219727 16259 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.195:39745 every 8 connection(s)
I20260429 14:08:24.221495 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 16129
I20260429 14:08:24.221624 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance
I20260429 14:08:24.221815 2839 external_mini_cluster.cc:1468] Setting key 7f59043a481e6d91ec39ded07a501063
Apr 29 14:08:24 dist-test-slave-8n77 krb5kdc[14989](info): TGS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777471703, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for HTTP/127.2.197.212@KRBTEST.COM
Apr 29 14:08:24 dist-test-slave-8n77 krb5kdc[14989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471704, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:08:24.235576 15982 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:24.220461 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:38355 (local address 127.2.197.254:40767)
0429 14:08:24.220510 (+ 49us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:24.220512 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:24.220939 (+ 427us) server_negotiation.cc:408] Connection header received
0429 14:08:24.222637 (+ 1698us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:24.222638 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:24.222654 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:24.222721 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:24.226406 (+ 3685us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:24.227245 (+ 839us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:24.228150 (+ 905us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:24.228329 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:24.232372 (+ 4043us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:24.232377 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:24.232378 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:24.232387 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:24.234123 (+ 1736us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:24.234655 (+ 532us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:24.234656 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:24.234657 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:24.234680 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:24.234889 (+ 209us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:24.234890 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:24.234890 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:24.235003 (+ 113us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:24.235049 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:24.235379 (+ 330us) server_negotiation.cc:300] Negotiation successful
0429 14:08:24.235478 (+ 99us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260429 14:08:24.236284 16262 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:24.220440 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40767 (local address 127.2.197.195:38355)
0429 14:08:24.220825 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:24.220860 (+ 35us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:24.222452 (+ 1592us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:24.222996 (+ 544us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:24.223001 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:24.223297 (+ 296us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:08:24.223697 (+ 400us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:24.223702 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:24.227392 (+ 3690us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:24.227393 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:24.228034 (+ 641us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:24.228037 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:24.228492 (+ 455us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:24.230484 (+ 1992us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:08:24.230499 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:08:24.232245 (+ 1746us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:08:24.234229 (+ 1984us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:24.234236 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:24.234239 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:24.234559 (+ 320us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:24.234745 (+ 186us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:24.234745 (+ 0us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:24.234746 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:24.234828 (+ 82us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:24.235077 (+ 249us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:08:24.235081 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:08:24.235232 (+ 151us) client_negotiation.cc:770] Sending connection context
0429 14:08:24.235925 (+ 693us) client_negotiation.cc:241] Negotiation successful
0429 14:08:24.236064 (+ 139us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":103,"threads_started":1}
I20260429 14:08:24.236941 16260 heartbeater.cc:344] Connected to a master server at 127.2.197.254:40767
I20260429 14:08:24.237051 16260 heartbeater.cc:461] Registering TS with master...
I20260429 14:08:24.237246 16260 heartbeater.cc:507] Master 127.2.197.254:40767 requested a full tablet report, sending...
I20260429 14:08:24.237663 15751 ts_manager.cc:194] Registered new tserver with Master: 039b92e0728d49828d1f6cad35a809e4 (127.2.197.195:39745)
I20260429 14:08:24.238119 15751 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.195@KRBTEST.COM'} at 127.2.197.195:38355
I20260429 14:08:24.244441 2839 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260429 14:08:24.245222 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 15847
I20260429 14:08:24.253756 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 15990
I20260429 14:08:24.261909 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 16129
I20260429 14:08:24.268707 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 15719
I20260429 14:08:24.288659 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:40767
--webserver_interface=127.2.197.254
--webserver_port=41761
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:40767
--ranger_config_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.197.212:39899/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.197.212:39899/kms with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:08:24.424072 16263 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:08:24.424341 16263 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:08:24.424387 16263 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:08:24.426978 16263 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:08:24.427066 16263 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:08:24.427089 16263 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:08:24.427107 16263 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:08:24.427125 16263 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:08:24.430059 16263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.197.212:39899/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:40767
--ranger_config_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:40767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=41761
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:08:24.430362 16263 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:08:24.430603 16263 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:08:24.433463 16268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:24.433563 16271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:24.433463 16269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:08:24.433909 16263 server_base.cc:1061] running on GCE node
I20260429 14:08:24.434139 16263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:08:24.434449 16263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:08:24.435652 16263 hybrid_clock.cc:648] HybridClock initialized: now 1777471704435630 us; error 36 us; skew 500 ppm
Apr 29 14:08:24 dist-test-slave-8n77 krb5kdc[14989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471704, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:08:24.438906 16263 init.cc:377] Logged in from keytab as kudu/127.2.197.254@KRBTEST.COM (short username kudu)
I20260429 14:08:24.439594 16263 webserver.cc:492] Webserver started at http://127.2.197.254:41761/ using document root <none> and password file <none>
I20260429 14:08:24.439841 16263 fs_manager.cc:362] Metadata directory not provided
I20260429 14:08:24.439913 16263 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
Apr 29 14:08:24 dist-test-slave-8n77 krb5kdc[14989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471704, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for HTTP/127.2.197.212@KRBTEST.COM
I20260429 14:08:24.519944 16263 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.000s sys 0.001s
I20260429 14:08:24.520686 16279 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:24.520889 16263 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:08:24.520943 16263 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "b9020c4422be4e73988c6bff8803a13e"
format_stamp: "Formatted at 2026-04-29 14:08:22 on dist-test-slave-8n77"
server_key: "67dbf21c9b9998539831bef285f0ca5c"
server_key_iv: "33976cf9ea813b02f96abb708f5cafdc"
server_key_version: "kuduclusterkey@0"
I20260429 14:08:24.521277 16263 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:08:24.543831 16263 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:08:24.546243 16263 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:08:24.546430 16263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:08:24.551223 16263 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:40767
I20260429 14:08:24.551330 16332 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:40767 every 8 connection(s)
I20260429 14:08:24.551774 16263 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:08:24.553123 16333 sys_catalog.cc:263] Verifying existing consensus state
I20260429 14:08:24.553853 16333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e: Bootstrap starting.
I20260429 14:08:24.557219 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 16263
I20260429 14:08:24.557541 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.197.212:39899/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.193:41755
--local_ip_for_outbound_sockets=127.2.197.193
--tserver_master_addrs=127.2.197.254:40767
--webserver_port=36745
--webserver_interface=127.2.197.193
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.197.212:39899/kms with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260429 14:08:24.564927 16333 log.cc:826] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e: Log is configured to *not* fsync() on all Append() calls
I20260429 14:08:24.566941 16333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260429 14:08:24.567272 16333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e: Bootstrap complete.
I20260429 14:08:24.570057 16333 raft_consensus.cc:359] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9020c4422be4e73988c6bff8803a13e" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40767 } }
I20260429 14:08:24.570339 16333 raft_consensus.cc:740] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9020c4422be4e73988c6bff8803a13e, State: Initialized, Role: FOLLOWER
I20260429 14:08:24.570580 16333 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9020c4422be4e73988c6bff8803a13e" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40767 } }
I20260429 14:08:24.570668 16333 raft_consensus.cc:399] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:08:24.570719 16333 raft_consensus.cc:493] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:08:24.570783 16333 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e [term 1 FOLLOWER]: Advancing to term 2
I20260429 14:08:24.572173 16333 raft_consensus.cc:515] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9020c4422be4e73988c6bff8803a13e" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40767 } }
I20260429 14:08:24.572351 16333 leader_election.cc:304] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9020c4422be4e73988c6bff8803a13e; no voters:
I20260429 14:08:24.572541 16333 leader_election.cc:290] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260429 14:08:24.572809 16333 sys_catalog.cc:565] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:08:24.574340 16333 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260429 14:08:24.574740 16338 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e [term 2 FOLLOWER]: Leader election won for term 2
I20260429 14:08:24.574898 16338 raft_consensus.cc:697] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e [term 2 LEADER]: Becoming Leader. State: Replica: b9020c4422be4e73988c6bff8803a13e, State: Running, Role: LEADER
I20260429 14:08:24.575009 16338 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9020c4422be4e73988c6bff8803a13e" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40767 } }
I20260429 14:08:24.575294 16338 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b9020c4422be4e73988c6bff8803a13e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9020c4422be4e73988c6bff8803a13e" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40767 } } }
I20260429 14:08:24.575393 16338 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e [sys.catalog]: This master's current role is: LEADER
I20260429 14:08:24.575585 16338 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9020c4422be4e73988c6bff8803a13e. Latest consensus state: current_term: 2 leader_uuid: "b9020c4422be4e73988c6bff8803a13e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9020c4422be4e73988c6bff8803a13e" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40767 } } }
I20260429 14:08:24.575775 16338 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e [sys.catalog]: This master's current role is: LEADER
I20260429 14:08:24.578688 16352 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:08:24.579458 16352 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:08:24.579638 16352 catalog_manager.cc:1269] Loaded cluster ID: 636bd676e72e47c695cdb2e901cff063
I20260429 14:08:24.579711 16352 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:08:24.581321 16352 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:08:24.581565 16352 catalog_manager.cc:6055] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e: Loaded TSK: 0
I20260429 14:08:24.581804 16352 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260429 14:08:24.680562 16335 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:08:24.680903 16335 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:08:24.680999 16335 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:08:24.683888 16335 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:08:24.684122 16335 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:08:24.684237 16335 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.193
I20260429 14:08:24.687361 16335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.197.212:39899/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.193:41755
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.197.193
--webserver_port=36745
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:40767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.193
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:08:24.688002 16335 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:08:24.688287 16335 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:08:24.691162 16379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:24.691520 16377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:24.692054 16376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:08:24.692921 16335 server_base.cc:1061] running on GCE node
I20260429 14:08:24.693203 16335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:08:24.693449 16335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:08:24.694629 16335 hybrid_clock.cc:648] HybridClock initialized: now 1777471704694602 us; error 42 us; skew 500 ppm
Apr 29 14:08:24 dist-test-slave-8n77 krb5kdc[14989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471704, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:08:24.697824 16335 init.cc:377] Logged in from keytab as kudu/127.2.197.193@KRBTEST.COM (short username kudu)
I20260429 14:08:24.698549 16335 webserver.cc:492] Webserver started at http://127.2.197.193:36745/ using document root <none> and password file <none>
I20260429 14:08:24.698803 16335 fs_manager.cc:362] Metadata directory not provided
I20260429 14:08:24.698872 16335 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
Apr 29 14:08:24 dist-test-slave-8n77 krb5kdc[14989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471704, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for HTTP/127.2.197.212@KRBTEST.COM
I20260429 14:08:24.742699 16335 fs_manager.cc:714] Time spent opening directory manager: real 0.012s user 0.001s sys 0.000s
I20260429 14:08:24.743458 16390 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:24.743662 16335 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260429 14:08:24.743722 16335 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-0/wal
uuid: "5631cd0a3f9448c2889a1b0872328418"
format_stamp: "Formatted at 2026-04-29 14:08:23 on dist-test-slave-8n77"
server_key: "505902956ed3d37899f7bed8cc982e8a"
server_key_iv: "f4333992a0b455918720a4c983a1cf08"
server_key_version: "kuduclusterkey@0"
I20260429 14:08:24.744087 16335 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:08:24.762785 16335 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:08:24.765180 16335 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:08:24.765383 16335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:08:24.765645 16335 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:08:24.766031 16335 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:08:24.766119 16335 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:24.766238 16335 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:08:24.766369 16335 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:24.774135 16335 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.193:41755
I20260429 14:08:24.774719 16335 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
I20260429 14:08:24.777416 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 16335
I20260429 14:08:24.777756 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.197.212:39899/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.194:43035
--local_ip_for_outbound_sockets=127.2.197.194
--tserver_master_addrs=127.2.197.254:40767
--webserver_port=34635
--webserver_interface=127.2.197.194
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.197.212:39899/kms with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260429 14:08:24.781128 16504 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.193:41755 every 8 connection(s)
Apr 29 14:08:24 dist-test-slave-8n77 krb5kdc[14989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471704, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:08:24.793408 16508 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:24.781882 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40767 (local address 127.2.197.193:47329)
0429 14:08:24.782236 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:24.782270 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:24.783385 (+ 1115us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:24.784108 (+ 723us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:24.784112 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:24.784398 (+ 286us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:08:24.784765 (+ 367us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:24.784769 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:24.786232 (+ 1463us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:24.786233 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:24.786881 (+ 648us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:24.786884 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:24.787019 (+ 135us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:24.787634 (+ 615us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:08:24.787645 (+ 11us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:08:24.789394 (+ 1749us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:08:24.791800 (+ 2406us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:24.791806 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:24.791809 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:24.792081 (+ 272us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:24.792304 (+ 223us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:24.792305 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:24.792305 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:24.792383 (+ 78us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:24.792797 (+ 414us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:08:24.792802 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:08:24.792939 (+ 137us) client_negotiation.cc:770] Sending connection context
0429 14:08:24.793045 (+ 106us) client_negotiation.cc:241] Negotiation successful
0429 14:08:24.793189 (+ 144us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":117,"threads_started":1}
I20260429 14:08:24.794445 16509 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:24.781913 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:47329 (local address 127.2.197.254:40767)
0429 14:08:24.783586 (+ 1673us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:24.783596 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:24.783620 (+ 24us) server_negotiation.cc:408] Connection header received
0429 14:08:24.783751 (+ 131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:24.783757 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:24.783804 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:24.783921 (+ 117us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:24.784893 (+ 972us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:24.786116 (+ 1223us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:24.788271 (+ 2155us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:24.788513 (+ 242us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:24.789523 (+ 1010us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:24.789548 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:24.789548 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:24.789562 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:24.791691 (+ 2129us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:24.792178 (+ 487us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:24.792181 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:24.792182 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:24.792233 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:24.792449 (+ 216us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:24.792450 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:24.792450 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:24.792679 (+ 229us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:24.792763 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:24.794087 (+ 1324us) server_negotiation.cc:300] Negotiation successful
0429 14:08:24.794208 (+ 121us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1629,"thread_start_us":90,"threads_started":1}
I20260429 14:08:24.795387 16505 heartbeater.cc:344] Connected to a master server at 127.2.197.254:40767
I20260429 14:08:24.795490 16505 heartbeater.cc:461] Registering TS with master...
I20260429 14:08:24.795719 16505 heartbeater.cc:507] Master 127.2.197.254:40767 requested a full tablet report, sending...
W20260429 14:08:24.796543 16505 heartbeater.cc:646] Failed to heartbeat to 127.2.197.254:40767 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260429 14:08:24.796612 16505 heartbeater.cc:461] Registering TS with master...
I20260429 14:08:24.796721 16505 heartbeater.cc:507] Master 127.2.197.254:40767 requested a full tablet report, sending...
I20260429 14:08:24.797147 16505 heartbeater.cc:461] Registering TS with master...
I20260429 14:08:24.797248 16505 heartbeater.cc:507] Master 127.2.197.254:40767 requested a full tablet report, sending...
W20260429 14:08:24.797588 16505 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260429 14:08:24.903190 16507 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:08:24.903541 16507 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:08:24.903607 16507 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:08:24.906052 16507 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:08:24.906176 16507 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:08:24.906257 16507 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.194
I20260429 14:08:24.908998 16507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.197.212:39899/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.194:43035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.197.194
--webserver_port=34635
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:40767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.194
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:08:24.909395 16507 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:08:24.909639 16507 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:08:24.915450 16517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:24.915864 16515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:24.916000 16514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:08:24.917339 16507 server_base.cc:1061] running on GCE node
I20260429 14:08:24.917702 16507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:08:24.917958 16507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:08:24.925323 16507 hybrid_clock.cc:648] HybridClock initialized: now 1777471704925285 us; error 55 us; skew 500 ppm
Apr 29 14:08:24 dist-test-slave-8n77 krb5kdc[14989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471704, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:08:24.928694 16507 init.cc:377] Logged in from keytab as kudu/127.2.197.194@KRBTEST.COM (short username kudu)
I20260429 14:08:24.929356 16507 webserver.cc:492] Webserver started at http://127.2.197.194:34635/ using document root <none> and password file <none>
I20260429 14:08:24.929600 16507 fs_manager.cc:362] Metadata directory not provided
I20260429 14:08:24.929664 16507 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
Apr 29 14:08:24 dist-test-slave-8n77 krb5kdc[14989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471704, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for HTTP/127.2.197.212@KRBTEST.COM
I20260429 14:08:24.963862 16507 fs_manager.cc:714] Time spent opening directory manager: real 0.012s user 0.001s sys 0.000s
I20260429 14:08:24.974059 16526 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:24.974382 16507 fs_manager.cc:730] Time spent opening block manager: real 0.010s user 0.000s sys 0.000s
I20260429 14:08:24.974470 16507 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-1/wal
uuid: "8bfed081c9264b66908c0cc446b51bf8"
format_stamp: "Formatted at 2026-04-29 14:08:23 on dist-test-slave-8n77"
server_key: "79daa690ad8b1dba5a1556e602a06424"
server_key_iv: "70a749d6c686f2825c714becd14c3770"
server_key_version: "kuduclusterkey@0"
I20260429 14:08:24.974851 16507 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:08:24.991055 16507 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:08:24.993489 16507 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:08:24.993706 16507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:08:24.993956 16507 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:08:24.994406 16507 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:08:24.994511 16507 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:24.994585 16507 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:08:24.994642 16507 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:25.001377 16507 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.194:43035
I20260429 14:08:25.002070 16507 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
I20260429 14:08:25.004734 16640 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.194:43035 every 8 connection(s)
I20260429 14:08:25.011798 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 16507
I20260429 14:08:25.012135 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.197.212:39899/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.195:39745
--local_ip_for_outbound_sockets=127.2.197.195
--tserver_master_addrs=127.2.197.254:40767
--webserver_port=40979
--webserver_interface=127.2.197.195
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.197.212:39899/kms with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Apr 29 14:08:25 dist-test-slave-8n77 krb5kdc[14989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471704, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:08:25.019387 16509 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:25.006378 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:33897 (local address 127.2.197.254:40767)
0429 14:08:25.006430 (+ 52us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:25.006433 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:25.006630 (+ 197us) server_negotiation.cc:408] Connection header received
0429 14:08:25.007827 (+ 1197us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:25.007828 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:25.007854 (+ 26us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:25.007885 (+ 31us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:25.008850 (+ 965us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:25.009625 (+ 775us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:25.010505 (+ 880us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:25.010706 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:25.015878 (+ 5172us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:25.015884 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:25.015884 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:25.015894 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:25.017790 (+ 1896us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:25.018386 (+ 596us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:25.018387 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:25.018388 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:25.018418 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:25.018644 (+ 226us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:25.018644 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:25.018645 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:25.018773 (+ 128us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:25.018822 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:25.019175 (+ 353us) server_negotiation.cc:300] Negotiation successful
0429 14:08:25.019290 (+ 115us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260429 14:08:25.020244 16643 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:25.006068 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40767 (local address 127.2.197.194:33897)
0429 14:08:25.006509 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:25.006549 (+ 40us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:25.007684 (+ 1135us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:25.008066 (+ 382us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:25.008071 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:25.008350 (+ 279us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:08:25.008731 (+ 381us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:25.008735 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:25.009769 (+ 1034us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:25.009770 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:25.010403 (+ 633us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:25.010405 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:25.010845 (+ 440us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:25.011456 (+ 611us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:08:25.011468 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:08:25.015731 (+ 4263us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:08:25.017924 (+ 2193us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:25.017934 (+ 10us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:25.017937 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:25.018257 (+ 320us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:25.018489 (+ 232us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:25.018490 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:25.018491 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:25.018585 (+ 94us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:25.018855 (+ 270us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:08:25.018860 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:08:25.019023 (+ 163us) client_negotiation.cc:770] Sending connection context
0429 14:08:25.019797 (+ 774us) client_negotiation.cc:241] Negotiation successful
0429 14:08:25.019960 (+ 163us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":91,"threads_started":1}
I20260429 14:08:25.021096 16641 heartbeater.cc:344] Connected to a master server at 127.2.197.254:40767
I20260429 14:08:25.021196 16641 heartbeater.cc:461] Registering TS with master...
I20260429 14:08:25.021437 16641 heartbeater.cc:507] Master 127.2.197.254:40767 requested a full tablet report, sending...
W20260429 14:08:25.022145 16641 heartbeater.cc:646] Failed to heartbeat to 127.2.197.254:40767 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260429 14:08:25.022214 16641 heartbeater.cc:461] Registering TS with master...
I20260429 14:08:25.022347 16641 heartbeater.cc:507] Master 127.2.197.254:40767 requested a full tablet report, sending...
I20260429 14:08:25.023022 16641 heartbeater.cc:461] Registering TS with master...
I20260429 14:08:25.023412 16641 heartbeater.cc:507] Master 127.2.197.254:40767 requested a full tablet report, sending...
W20260429 14:08:25.023993 16641 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260429 14:08:25.132198 16644 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:08:25.132547 16644 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:08:25.132637 16644 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:08:25.135342 16644 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:08:25.135491 16644 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:08:25.135593 16644 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.195
I20260429 14:08:25.138525 16644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.197.212:39899/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.195:39745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.197.195
--webserver_port=40979
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:40767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.195
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:08:25.139029 16644 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:08:25.139294 16644 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:08:25.142215 16652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:25.142346 16650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:25.142554 16649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:08:25.143538 16644 server_base.cc:1061] running on GCE node
I20260429 14:08:25.143729 16644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:08:25.143970 16644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:08:25.145147 16644 hybrid_clock.cc:648] HybridClock initialized: now 1777471705145131 us; error 43 us; skew 500 ppm
Apr 29 14:08:25 dist-test-slave-8n77 krb5kdc[14989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471705, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:08:25.148689 16644 init.cc:377] Logged in from keytab as kudu/127.2.197.195@KRBTEST.COM (short username kudu)
I20260429 14:08:25.149472 16644 webserver.cc:492] Webserver started at http://127.2.197.195:40979/ using document root <none> and password file <none>
I20260429 14:08:25.149703 16644 fs_manager.cc:362] Metadata directory not provided
I20260429 14:08:25.149785 16644 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
Apr 29 14:08:25 dist-test-slave-8n77 krb5kdc[14989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471705, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for HTTP/127.2.197.212@KRBTEST.COM
I20260429 14:08:25.165307 16644 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:08:25.166069 16661 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:25.166342 16644 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:08:25.166432 16644 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-2/wal
uuid: "039b92e0728d49828d1f6cad35a809e4"
format_stamp: "Formatted at 2026-04-29 14:08:24 on dist-test-slave-8n77"
server_key: "55732e10623447bbc613f4fa507a3a49"
server_key_iv: "b2f4f47ab8b8d4325b33c5f471d35623"
server_key_version: "kuduclusterkey@0"
I20260429 14:08:25.166807 16644 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:08:25.187896 16644 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:08:25.190326 16644 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:08:25.190546 16644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:08:25.190836 16644 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:08:25.191236 16644 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:08:25.191321 16644 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:25.191395 16644 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:08:25.191468 16644 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:25.198993 16644 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.195:39745
I20260429 14:08:25.199710 16644 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
I20260429 14:08:25.202373 16774 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.195:39745 every 8 connection(s)
I20260429 14:08:25.209457 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 16644
Apr 29 14:08:25 dist-test-slave-8n77 krb5kdc[14989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471705, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:08:25.213698 16509 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:25.203376 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:57823 (local address 127.2.197.254:40767)
0429 14:08:25.203432 (+ 56us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:25.203435 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:25.203715 (+ 280us) server_negotiation.cc:408] Connection header received
0429 14:08:25.204871 (+ 1156us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:25.204873 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:25.204889 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:25.204922 (+ 33us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:25.205914 (+ 992us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:25.206729 (+ 815us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:25.207531 (+ 802us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:25.207709 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:25.210377 (+ 2668us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:25.210381 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:25.210382 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:25.210391 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:25.212152 (+ 1761us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:25.212682 (+ 530us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:25.212683 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:25.212684 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:25.212712 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:25.212962 (+ 250us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:25.212963 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:25.212964 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:25.213087 (+ 123us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:25.213129 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:25.213493 (+ 364us) server_negotiation.cc:300] Negotiation successful
0429 14:08:25.213602 (+ 109us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260429 14:08:25.213805 16777 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:25.203358 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40767 (local address 127.2.197.195:57823)
0429 14:08:25.203550 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:25.203589 (+ 39us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:25.204731 (+ 1142us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:25.205107 (+ 376us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:25.205112 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:25.205395 (+ 283us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:08:25.205786 (+ 391us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:25.205791 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:25.206835 (+ 1044us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:25.206836 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:25.207422 (+ 586us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:25.207425 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:25.207846 (+ 421us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:25.208641 (+ 795us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:08:25.208654 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:08:25.210234 (+ 1580us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:08:25.212258 (+ 2024us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:25.212266 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:25.212269 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:25.212583 (+ 314us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:25.212793 (+ 210us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:25.212793 (+ 0us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:25.212794 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:25.212879 (+ 85us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:25.213172 (+ 293us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:08:25.213177 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:08:25.213328 (+ 151us) client_negotiation.cc:770] Sending connection context
0429 14:08:25.213445 (+ 117us) client_negotiation.cc:241] Negotiation successful
0429 14:08:25.213581 (+ 136us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":96,"threads_started":1}
I20260429 14:08:25.214361 16775 heartbeater.cc:344] Connected to a master server at 127.2.197.254:40767
I20260429 14:08:25.214458 16775 heartbeater.cc:461] Registering TS with master...
I20260429 14:08:25.214665 16775 heartbeater.cc:507] Master 127.2.197.254:40767 requested a full tablet report, sending...
W20260429 14:08:25.215237 16775 heartbeater.cc:646] Failed to heartbeat to 127.2.197.254:40767 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260429 14:08:25.215297 16775 heartbeater.cc:461] Registering TS with master...
I20260429 14:08:25.215377 16775 heartbeater.cc:507] Master 127.2.197.254:40767 requested a full tablet report, sending...
I20260429 14:08:25.215683 16775 heartbeater.cc:461] Registering TS with master...
I20260429 14:08:25.215786 16775 heartbeater.cc:507] Master 127.2.197.254:40767 requested a full tablet report, sending...
W20260429 14:08:25.216068 16775 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260429 14:08:25.223372 16509 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:25.210079 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:39748 (local address 127.2.197.254:40767)
0429 14:08:25.213790 (+ 3711us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:25.213792 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:25.216149 (+ 2357us) server_negotiation.cc:408] Connection header received
0429 14:08:25.216172 (+ 23us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:25.216173 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:25.216183 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:25.216237 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:25.217200 (+ 963us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:25.217934 (+ 734us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:25.218781 (+ 847us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:25.218936 (+ 155us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:25.220283 (+ 1347us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:25.220287 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:25.220287 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:25.220295 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:25.222111 (+ 1816us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:25.222587 (+ 476us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:25.222589 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:25.222590 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:25.222618 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:25.222821 (+ 203us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:25.222822 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:25.222822 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:25.222937 (+ 115us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:25.223082 (+ 145us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:25.223179 (+ 97us) server_negotiation.cc:300] Negotiation successful
0429 14:08:25.223273 (+ 94us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3691,"thread_start_us":57,"threads_started":1}
I20260429 14:08:25.797777 16505 heartbeater.cc:461] Registering TS with master...
I20260429 14:08:25.797978 16505 heartbeater.cc:507] Master 127.2.197.254:40767 requested a full tablet report, sending...
I20260429 14:08:25.798475 16296 ts_manager.cc:194] Registered new tserver with Master: 5631cd0a3f9448c2889a1b0872328418 (127.2.197.193:41755)
I20260429 14:08:25.799499 16296 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.193@KRBTEST.COM'} at 127.2.197.193:47329
Apr 29 14:08:25 dist-test-slave-8n77 krb5kdc[14989](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777471705, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:08:26.024161 16641 heartbeater.cc:461] Registering TS with master...
I20260429 14:08:26.024394 16641 heartbeater.cc:507] Master 127.2.197.254:40767 requested a full tablet report, sending...
I20260429 14:08:26.024747 16296 ts_manager.cc:194] Registered new tserver with Master: 8bfed081c9264b66908c0cc446b51bf8 (127.2.197.194:43035)
I20260429 14:08:26.025174 16296 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.194@KRBTEST.COM'} at 127.2.197.194:33897
I20260429 14:08:26.216223 16775 heartbeater.cc:461] Registering TS with master...
I20260429 14:08:26.216444 16775 heartbeater.cc:507] Master 127.2.197.254:40767 requested a full tablet report, sending...
I20260429 14:08:26.216811 16296 ts_manager.cc:194] Registered new tserver with Master: 039b92e0728d49828d1f6cad35a809e4 (127.2.197.195:39745)
I20260429 14:08:26.217355 16296 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.195@KRBTEST.COM'} at 127.2.197.195:57823
I20260429 14:08:26.221982 2839 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260429 14:08:26.233629 16801 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:26.225388 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:39752 (local address 127.2.197.254:40767)
0429 14:08:26.225929 (+ 541us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:26.225931 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:26.226304 (+ 373us) server_negotiation.cc:408] Connection header received
0429 14:08:26.226441 (+ 137us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:26.226442 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:26.226456 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:26.226499 (+ 43us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:26.227529 (+ 1030us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:26.228368 (+ 839us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:26.229199 (+ 831us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:26.229375 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:26.230836 (+ 1461us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:26.230841 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:26.230842 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:26.230850 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:26.232476 (+ 1626us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:26.232876 (+ 400us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:26.232877 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:26.232878 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:26.232908 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:26.233064 (+ 156us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:26.233066 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:26.233066 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:26.233166 (+ 100us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:26.233273 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:26.233390 (+ 117us) server_negotiation.cc:300] Negotiation successful
0429 14:08:26.233490 (+ 100us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":90,"threads_started":1}
I20260429 14:08:26.235605 16296 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39752:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 14:08:26.237406 16296 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 14:08:26.249552 16808 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:26.241329 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:43035 (local address 127.0.0.1:36738)
0429 14:08:26.244918 (+ 3589us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:26.244928 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:26.244979 (+ 51us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:26.246382 (+ 1403us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:26.246383 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:26.246385 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:08:26.246567 (+ 182us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:26.246569 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:26.248345 (+ 1776us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:26.248346 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:26.249334 (+ 988us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:26.249337 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:26.249397 (+ 60us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:26.249402 (+ 5us) client_negotiation.cc:770] Sending connection context
0429 14:08:26.249433 (+ 31us) client_negotiation.cc:241] Negotiation successful
0429 14:08:26.249470 (+ 37us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3572,"thread_start_us":70,"threads_started":1}
I20260429 14:08:26.250620 16813 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:26.241754 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36738 (local address 127.2.197.194:43035)
0429 14:08:26.242439 (+ 685us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:26.242442 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:26.245039 (+ 2597us) server_negotiation.cc:408] Connection header received
0429 14:08:26.245079 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:26.245081 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:26.245097 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:26.245154 (+ 57us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:08:26.247420 (+ 2266us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:26.248253 (+ 833us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:26.249895 (+ 1642us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:26.250417 (+ 522us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:26.250478 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:26.250515 (+ 37us) server_negotiation.cc:300] Negotiation successful
0429 14:08:26.250554 (+ 39us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":50,"threads_started":1}
I20260429 14:08:26.250696 16812 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:26.241634 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:45120 (local address 127.2.197.195:39745)
0429 14:08:26.242372 (+ 738us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:26.242377 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:26.242632 (+ 255us) server_negotiation.cc:408] Connection header received
0429 14:08:26.242682 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:26.242684 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:26.242701 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:26.242761 (+ 60us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:08:26.243316 (+ 555us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:26.244786 (+ 1470us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:26.249797 (+ 5011us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:26.250471 (+ 674us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:26.250509 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:26.250539 (+ 30us) server_negotiation.cc:300] Negotiation successful
0429 14:08:26.250579 (+ 40us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":56,"threads_started":1}
I20260429 14:08:26.251502 16809 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:26.240994 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:45424 (local address 127.2.197.193:41755)
0429 14:08:26.245289 (+ 4295us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:26.245293 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:26.245302 (+ 9us) server_negotiation.cc:408] Connection header received
0429 14:08:26.245338 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:26.245340 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:26.245354 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:26.245427 (+ 73us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:08:26.246678 (+ 1251us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:26.247766 (+ 1088us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:26.251020 (+ 3254us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:26.251371 (+ 351us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:26.251401 (+ 30us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:26.251427 (+ 26us) server_negotiation.cc:300] Negotiation successful
0429 14:08:26.251450 (+ 23us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4250,"thread_start_us":52,"threads_started":1}
I20260429 14:08:26.249734 16811 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:26.240767 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:41755 (local address 127.0.0.1:45424)
0429 14:08:26.242165 (+ 1398us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:26.242196 (+ 31us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:26.242262 (+ 66us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:26.245512 (+ 3250us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:26.245513 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:26.245516 (+ 3us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:08:26.245732 (+ 216us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:26.245734 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:26.248370 (+ 2636us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:26.248371 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:26.249538 (+ 1167us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:26.249540 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:26.249595 (+ 55us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:26.249598 (+ 3us) client_negotiation.cc:770] Sending connection context
0429 14:08:26.249629 (+ 31us) client_negotiation.cc:241] Negotiation successful
0429 14:08:26.249666 (+ 37us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1370,"thread_start_us":66,"threads_started":1}
I20260429 14:08:26.252475 16438 tablet_service.cc:1511] Processing CreateTablet for tablet 5ec8ca2f9c6f48e7b5f0756c8d7055a2 (DEFAULT_TABLE table=test-table [id=2dbf9520cb68418793288ad19d6f1cc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:08:26.252527 16575 tablet_service.cc:1511] Processing CreateTablet for tablet 5ec8ca2f9c6f48e7b5f0756c8d7055a2 (DEFAULT_TABLE table=test-table [id=2dbf9520cb68418793288ad19d6f1cc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:08:26.252779 16575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ec8ca2f9c6f48e7b5f0756c8d7055a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:08:26.252779 16438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ec8ca2f9c6f48e7b5f0756c8d7055a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:08:26.250373 16810 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:26.241166 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:39745 (local address 127.0.0.1:45120)
0429 14:08:26.242527 (+ 1361us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:26.242535 (+ 8us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:26.242580 (+ 45us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:26.242902 (+ 322us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:26.242905 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:26.242908 (+ 3us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:08:26.243203 (+ 295us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:26.243206 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:26.245824 (+ 2618us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:26.245825 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:26.247084 (+ 1259us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:26.247087 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:26.247152 (+ 65us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:26.247220 (+ 68us) client_negotiation.cc:770] Sending connection context
0429 14:08:26.247271 (+ 51us) client_negotiation.cc:241] Negotiation successful
0429 14:08:26.247315 (+ 44us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1347,"thread_start_us":78,"threads_started":1}
I20260429 14:08:26.256608 16815 tablet_bootstrap.cc:492] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 8bfed081c9264b66908c0cc446b51bf8: Bootstrap starting.
I20260429 14:08:26.257036 16709 tablet_service.cc:1511] Processing CreateTablet for tablet 5ec8ca2f9c6f48e7b5f0756c8d7055a2 (DEFAULT_TABLE table=test-table [id=2dbf9520cb68418793288ad19d6f1cc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:08:26.257275 16815 tablet_bootstrap.cc:654] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 8bfed081c9264b66908c0cc446b51bf8: Neither blocks nor log segments found. Creating new log.
I20260429 14:08:26.257297 16709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ec8ca2f9c6f48e7b5f0756c8d7055a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:08:26.257557 16815 log.cc:826] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 8bfed081c9264b66908c0cc446b51bf8: Log is configured to *not* fsync() on all Append() calls
I20260429 14:08:26.257858 16814 tablet_bootstrap.cc:492] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 5631cd0a3f9448c2889a1b0872328418: Bootstrap starting.
I20260429 14:08:26.258451 16814 tablet_bootstrap.cc:654] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 5631cd0a3f9448c2889a1b0872328418: Neither blocks nor log segments found. Creating new log.
I20260429 14:08:26.259109 16814 log.cc:826] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 5631cd0a3f9448c2889a1b0872328418: Log is configured to *not* fsync() on all Append() calls
I20260429 14:08:26.259826 16815 tablet_bootstrap.cc:492] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 8bfed081c9264b66908c0cc446b51bf8: No bootstrap required, opened a new log
I20260429 14:08:26.259882 16815 ts_tablet_manager.cc:1403] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 8bfed081c9264b66908c0cc446b51bf8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260429 14:08:26.261224 16818 tablet_bootstrap.cc:492] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 039b92e0728d49828d1f6cad35a809e4: Bootstrap starting.
I20260429 14:08:26.262070 16815 raft_consensus.cc:359] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 8bfed081c9264b66908c0cc446b51bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5631cd0a3f9448c2889a1b0872328418" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41755 } } peers { permanent_uuid: "039b92e0728d49828d1f6cad35a809e4" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 39745 } } peers { permanent_uuid: "8bfed081c9264b66908c0cc446b51bf8" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 43035 } }
I20260429 14:08:26.262385 16815 raft_consensus.cc:385] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 8bfed081c9264b66908c0cc446b51bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:08:26.262490 16815 raft_consensus.cc:740] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 8bfed081c9264b66908c0cc446b51bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bfed081c9264b66908c0cc446b51bf8, State: Initialized, Role: FOLLOWER
I20260429 14:08:26.262669 16815 consensus_queue.cc:260] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 8bfed081c9264b66908c0cc446b51bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5631cd0a3f9448c2889a1b0872328418" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41755 } } peers { permanent_uuid: "039b92e0728d49828d1f6cad35a809e4" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 39745 } } peers { permanent_uuid: "8bfed081c9264b66908c0cc446b51bf8" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 43035 } }
I20260429 14:08:26.263039 16815 ts_tablet_manager.cc:1434] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 8bfed081c9264b66908c0cc446b51bf8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260429 14:08:26.263213 16641 heartbeater.cc:499] Master 127.2.197.254:40767 was elected leader, sending a full tablet report...
I20260429 14:08:26.265077 16814 tablet_bootstrap.cc:492] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 5631cd0a3f9448c2889a1b0872328418: No bootstrap required, opened a new log
I20260429 14:08:26.265210 16814 ts_tablet_manager.cc:1403] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 5631cd0a3f9448c2889a1b0872328418: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20260429 14:08:26.266655 16818 tablet_bootstrap.cc:654] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 039b92e0728d49828d1f6cad35a809e4: Neither blocks nor log segments found. Creating new log.
I20260429 14:08:26.267004 16818 log.cc:826] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 039b92e0728d49828d1f6cad35a809e4: Log is configured to *not* fsync() on all Append() calls
I20260429 14:08:26.267735 16814 raft_consensus.cc:359] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 5631cd0a3f9448c2889a1b0872328418 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5631cd0a3f9448c2889a1b0872328418" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41755 } } peers { permanent_uuid: "039b92e0728d49828d1f6cad35a809e4" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 39745 } } peers { permanent_uuid: "8bfed081c9264b66908c0cc446b51bf8" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 43035 } }
I20260429 14:08:26.267880 16814 raft_consensus.cc:385] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 5631cd0a3f9448c2889a1b0872328418 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:08:26.267939 16814 raft_consensus.cc:740] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 5631cd0a3f9448c2889a1b0872328418 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5631cd0a3f9448c2889a1b0872328418, State: Initialized, Role: FOLLOWER
I20260429 14:08:26.268064 16814 consensus_queue.cc:260] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 5631cd0a3f9448c2889a1b0872328418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5631cd0a3f9448c2889a1b0872328418" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41755 } } peers { permanent_uuid: "039b92e0728d49828d1f6cad35a809e4" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 39745 } } peers { permanent_uuid: "8bfed081c9264b66908c0cc446b51bf8" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 43035 } }
I20260429 14:08:26.268337 16814 ts_tablet_manager.cc:1434] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 5631cd0a3f9448c2889a1b0872328418: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260429 14:08:26.268375 16505 heartbeater.cc:499] Master 127.2.197.254:40767 was elected leader, sending a full tablet report...
I20260429 14:08:26.268702 16819 raft_consensus.cc:493] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 8bfed081c9264b66908c0cc446b51bf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:08:26.268776 16819 raft_consensus.cc:515] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 8bfed081c9264b66908c0cc446b51bf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5631cd0a3f9448c2889a1b0872328418" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41755 } } peers { permanent_uuid: "039b92e0728d49828d1f6cad35a809e4" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 39745 } } peers { permanent_uuid: "8bfed081c9264b66908c0cc446b51bf8" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 43035 } }
I20260429 14:08:26.269761 16819 leader_election.cc:290] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 8bfed081c9264b66908c0cc446b51bf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5631cd0a3f9448c2889a1b0872328418 (127.2.197.193:41755), 039b92e0728d49828d1f6cad35a809e4 (127.2.197.195:39745)
I20260429 14:08:26.273160 16809 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:26.269337 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:50141 (local address 127.2.197.193:41755)
0429 14:08:26.269379 (+ 42us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:26.269382 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:26.269815 (+ 433us) server_negotiation.cc:408] Connection header received
0429 14:08:26.269920 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:26.269922 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:26.269934 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:26.269964 (+ 30us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:08:26.270466 (+ 502us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:26.271214 (+ 748us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:26.272454 (+ 1240us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:26.272910 (+ 456us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:26.272919 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:26.273037 (+ 118us) server_negotiation.cc:300] Negotiation successful
0429 14:08:26.273072 (+ 35us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260429 14:08:26.273380 16812 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:26.269699 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:50295 (local address 127.2.197.195:39745)
0429 14:08:26.269740 (+ 41us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:26.269743 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:26.269973 (+ 230us) server_negotiation.cc:408] Connection header received
0429 14:08:26.270072 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:26.270074 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:26.270083 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:26.270114 (+ 31us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:08:26.270893 (+ 779us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:26.271625 (+ 732us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:26.272647 (+ 1022us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:26.273125 (+ 478us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:26.273133 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:26.273267 (+ 134us) server_negotiation.cc:300] Negotiation successful
0429 14:08:26.273315 (+ 48us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260429 14:08:26.273671 16823 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:26.269518 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:39745 (local address 127.2.197.194:50295)
0429 14:08:26.269914 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:26.269923 (+ 9us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:26.269996 (+ 73us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:26.270187 (+ 191us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:26.270188 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:26.270189 (+ 1us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:08:26.270805 (+ 616us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:26.270809 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:26.271713 (+ 904us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:26.271714 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:26.272561 (+ 847us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:26.272563 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:26.273176 (+ 613us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:26.273179 (+ 3us) client_negotiation.cc:770] Sending connection context
0429 14:08:26.273550 (+ 371us) client_negotiation.cc:241] Negotiation successful
0429 14:08:26.273579 (+ 29us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":66,"threads_started":1}
I20260429 14:08:26.273677 16824 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:26.269160 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:41755 (local address 127.2.197.194:50141)
0429 14:08:26.269752 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:26.269764 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:26.269844 (+ 80us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:26.270047 (+ 203us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:26.270048 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:26.270050 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:08:26.270378 (+ 328us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:26.270380 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:26.271305 (+ 925us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:26.271305 (+ 0us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:26.272359 (+ 1054us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:26.272361 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:26.272949 (+ 588us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:26.272952 (+ 3us) client_negotiation.cc:770] Sending connection context
0429 14:08:26.273470 (+ 518us) client_negotiation.cc:241] Negotiation successful
0429 14:08:26.273577 (+ 107us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":86,"threads_started":1}
I20260429 14:08:26.274106 16726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ec8ca2f9c6f48e7b5f0756c8d7055a2" candidate_uuid: "8bfed081c9264b66908c0cc446b51bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039b92e0728d49828d1f6cad35a809e4" is_pre_election: true
W20260429 14:08:26.275447 16528 leader_election.cc:343] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 8bfed081c9264b66908c0cc446b51bf8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 039b92e0728d49828d1f6cad35a809e4 (127.2.197.195:39745): Illegal state: must be running to vote when last-logged opid is not known
I20260429 14:08:26.275692 16818 tablet_bootstrap.cc:492] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 039b92e0728d49828d1f6cad35a809e4: No bootstrap required, opened a new log
I20260429 14:08:26.275787 16818 ts_tablet_manager.cc:1403] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 039b92e0728d49828d1f6cad35a809e4: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.000s
I20260429 14:08:26.276531 16818 raft_consensus.cc:359] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 039b92e0728d49828d1f6cad35a809e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5631cd0a3f9448c2889a1b0872328418" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41755 } } peers { permanent_uuid: "039b92e0728d49828d1f6cad35a809e4" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 39745 } } peers { permanent_uuid: "8bfed081c9264b66908c0cc446b51bf8" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 43035 } }
I20260429 14:08:26.276669 16818 raft_consensus.cc:385] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 039b92e0728d49828d1f6cad35a809e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:08:26.276710 16818 raft_consensus.cc:740] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 039b92e0728d49828d1f6cad35a809e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039b92e0728d49828d1f6cad35a809e4, State: Initialized, Role: FOLLOWER
I20260429 14:08:26.276814 16818 consensus_queue.cc:260] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 039b92e0728d49828d1f6cad35a809e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5631cd0a3f9448c2889a1b0872328418" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41755 } } peers { permanent_uuid: "039b92e0728d49828d1f6cad35a809e4" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 39745 } } peers { permanent_uuid: "8bfed081c9264b66908c0cc446b51bf8" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 43035 } }
I20260429 14:08:26.277070 16818 ts_tablet_manager.cc:1434] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 039b92e0728d49828d1f6cad35a809e4: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260429 14:08:26.277082 16775 heartbeater.cc:499] Master 127.2.197.254:40767 was elected leader, sending a full tablet report...
I20260429 14:08:26.277500 16455 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ec8ca2f9c6f48e7b5f0756c8d7055a2" candidate_uuid: "8bfed081c9264b66908c0cc446b51bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5631cd0a3f9448c2889a1b0872328418" is_pre_election: true
I20260429 14:08:26.277638 16455 raft_consensus.cc:2468] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 5631cd0a3f9448c2889a1b0872328418 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bfed081c9264b66908c0cc446b51bf8 in term 0.
I20260429 14:08:26.277858 16530 leader_election.cc:304] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 8bfed081c9264b66908c0cc446b51bf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5631cd0a3f9448c2889a1b0872328418, 8bfed081c9264b66908c0cc446b51bf8; no voters: 039b92e0728d49828d1f6cad35a809e4
I20260429 14:08:26.278098 16820 raft_consensus.cc:2804] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 8bfed081c9264b66908c0cc446b51bf8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 14:08:26.278157 16820 raft_consensus.cc:493] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 8bfed081c9264b66908c0cc446b51bf8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 14:08:26.278203 16820 raft_consensus.cc:3060] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 8bfed081c9264b66908c0cc446b51bf8 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:08:26.279011 16820 raft_consensus.cc:515] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 8bfed081c9264b66908c0cc446b51bf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5631cd0a3f9448c2889a1b0872328418" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41755 } } peers { permanent_uuid: "039b92e0728d49828d1f6cad35a809e4" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 39745 } } peers { permanent_uuid: "8bfed081c9264b66908c0cc446b51bf8" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 43035 } }
I20260429 14:08:26.279414 16455 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ec8ca2f9c6f48e7b5f0756c8d7055a2" candidate_uuid: "8bfed081c9264b66908c0cc446b51bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5631cd0a3f9448c2889a1b0872328418"
I20260429 14:08:26.279409 16726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ec8ca2f9c6f48e7b5f0756c8d7055a2" candidate_uuid: "8bfed081c9264b66908c0cc446b51bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039b92e0728d49828d1f6cad35a809e4"
I20260429 14:08:26.279490 16726 raft_consensus.cc:3060] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 039b92e0728d49828d1f6cad35a809e4 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:08:26.279490 16455 raft_consensus.cc:3060] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 5631cd0a3f9448c2889a1b0872328418 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:08:26.280335 16455 raft_consensus.cc:2468] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 5631cd0a3f9448c2889a1b0872328418 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bfed081c9264b66908c0cc446b51bf8 in term 1.
I20260429 14:08:26.280347 16726 raft_consensus.cc:2468] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 039b92e0728d49828d1f6cad35a809e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bfed081c9264b66908c0cc446b51bf8 in term 1.
I20260429 14:08:26.280510 16528 leader_election.cc:304] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 8bfed081c9264b66908c0cc446b51bf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039b92e0728d49828d1f6cad35a809e4, 8bfed081c9264b66908c0cc446b51bf8; no voters:
I20260429 14:08:26.280586 16819 raft_consensus.cc:2804] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 8bfed081c9264b66908c0cc446b51bf8 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:08:26.280687 16820 leader_election.cc:290] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 8bfed081c9264b66908c0cc446b51bf8 [CANDIDATE]: Term 1 election: Requested vote from peers 5631cd0a3f9448c2889a1b0872328418 (127.2.197.193:41755), 039b92e0728d49828d1f6cad35a809e4 (127.2.197.195:39745)
I20260429 14:08:26.280737 16819 raft_consensus.cc:697] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 8bfed081c9264b66908c0cc446b51bf8 [term 1 LEADER]: Becoming Leader. State: Replica: 8bfed081c9264b66908c0cc446b51bf8, State: Running, Role: LEADER
I20260429 14:08:26.280822 16819 consensus_queue.cc:237] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 8bfed081c9264b66908c0cc446b51bf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5631cd0a3f9448c2889a1b0872328418" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41755 } } peers { permanent_uuid: "039b92e0728d49828d1f6cad35a809e4" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 39745 } } peers { permanent_uuid: "8bfed081c9264b66908c0cc446b51bf8" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 43035 } }
I20260429 14:08:26.281453 16297 catalog_manager.cc:5671] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 8bfed081c9264b66908c0cc446b51bf8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8bfed081c9264b66908c0cc446b51bf8 (127.2.197.194). New cstate: current_term: 1 leader_uuid: "8bfed081c9264b66908c0cc446b51bf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5631cd0a3f9448c2889a1b0872328418" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039b92e0728d49828d1f6cad35a809e4" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 39745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8bfed081c9264b66908c0cc446b51bf8" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 43035 } health_report { overall_health: HEALTHY } } }
W20260429 14:08:26.283041 16506 tablet.cc:2404] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 5631cd0a3f9448c2889a1b0872328418: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 14:08:26.293505 16813 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:26.290596 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36754 (local address 127.2.197.194:43035)
0429 14:08:26.290644 (+ 48us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:26.290646 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:26.290655 (+ 9us) server_negotiation.cc:408] Connection header received
0429 14:08:26.290701 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:26.290702 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:26.290720 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:26.290776 (+ 56us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:08:26.291143 (+ 367us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:26.291855 (+ 712us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:26.292721 (+ 866us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:26.292919 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:26.292973 (+ 54us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:08:26.293086 (+ 113us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:08:26.293157 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:26.293397 (+ 240us) server_negotiation.cc:300] Negotiation successful
0429 14:08:26.293424 (+ 27us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260429 14:08:26.304541 16726 raft_consensus.cc:1275] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 039b92e0728d49828d1f6cad35a809e4 [term 1 FOLLOWER]: Refusing update from remote peer 8bfed081c9264b66908c0cc446b51bf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:08:26.304989 16826 consensus_queue.cc:1048] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 8bfed081c9264b66908c0cc446b51bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039b92e0728d49828d1f6cad35a809e4" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 39745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:08:26.305565 16455 raft_consensus.cc:1275] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 5631cd0a3f9448c2889a1b0872328418 [term 1 FOLLOWER]: Refusing update from remote peer 8bfed081c9264b66908c0cc446b51bf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:08:26.305882 16819 consensus_queue.cc:1048] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 8bfed081c9264b66908c0cc446b51bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5631cd0a3f9448c2889a1b0872328418" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 41755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:08:26.308764 16830 mvcc.cc:204] Tried to move back new op lower bound from 7280524109019983872 to 7280524108926709760. Current Snapshot: MvccSnapshot[applied={T|T < 7280524109019983872}]
I20260429 14:08:26.310947 16832 mvcc.cc:204] Tried to move back new op lower bound from 7280524109019983872 to 7280524108926709760. Current Snapshot: MvccSnapshot[applied={T|T < 7280524109019983872}]
I20260429 14:08:26.311935 16297 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:08:26.312052 16297 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:08:26.312892 16297 catalog_manager.cc:5958] T 00000000000000000000000000000000 P b9020c4422be4e73988c6bff8803a13e: Sending DeleteTablet for 3 replicas of tablet 5ec8ca2f9c6f48e7b5f0756c8d7055a2
I20260429 14:08:26.313411 16709 tablet_service.cc:1558] Processing DeleteTablet for tablet 5ec8ca2f9c6f48e7b5f0756c8d7055a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:08:26 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:45120
I20260429 14:08:26.313638 16575 tablet_service.cc:1558] Processing DeleteTablet for tablet 5ec8ca2f9c6f48e7b5f0756c8d7055a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:08:26 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:36738
I20260429 14:08:26.314144 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 16335
I20260429 14:08:26.322410 16838 tablet_replica.cc:333] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 039b92e0728d49828d1f6cad35a809e4: stopping tablet replica
I20260429 14:08:26.322639 16838 raft_consensus.cc:2243] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 039b92e0728d49828d1f6cad35a809e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:08:26.322896 16838 raft_consensus.cc:2272] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 039b92e0728d49828d1f6cad35a809e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:08:26.323125 16839 tablet_replica.cc:333] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 8bfed081c9264b66908c0cc446b51bf8: stopping tablet replica
I20260429 14:08:26.323256 16839 raft_consensus.cc:2243] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 8bfed081c9264b66908c0cc446b51bf8 [term 1 LEADER]: Raft consensus shutting down.
W20260429 14:08:26.323345 16283 connection.cc:570] client connection to 127.2.197.193:41755 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260429 14:08:26.323446 16283 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260429 14:08:26.323421 16839 raft_consensus.cc:2272] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 8bfed081c9264b66908c0cc446b51bf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:08:26.324065 16839 ts_tablet_manager.cc:1916] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 8bfed081c9264b66908c0cc446b51bf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:08:26.324088 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 16507
I20260429 14:08:26.331236 16810 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:26.331013 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:41755 (local address 127.0.0.1:45432)
0429 14:08:26.331108 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:26.331189 (+ 81us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.193:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260429 14:08:26.331404 16283 catalog_manager.cc:4729] TS 5631cd0a3f9448c2889a1b0872328418 (127.2.197.193:41755): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5ec8ca2f9c6f48e7b5f0756c8d7055a2: Network error: Client connection negotiation failed: client connection to 127.2.197.193:41755: connect: Connection refused (error 111)
I20260429 14:08:26.331859 16838 ts_tablet_manager.cc:1916] T 5ec8ca2f9c6f48e7b5f0756c8d7055a2 P 039b92e0728d49828d1f6cad35a809e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:08:26.332618 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 16644
I20260429 14:08:26.332777 16810 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:26.332685 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:43035 (local address 127.0.0.1:36760)
0429 14:08:26.332728 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:26.332749 (+ 21us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.194:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260429 14:08:26.338129 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 16263
I20260429 14:08:26.338172 16810 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:26.338060 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:39745 (local address 127.0.0.1:45128)
0429 14:08:26.338110 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:26.338132 (+ 22us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.195:39745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260429 14:08:26.344558 2839 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260429 14:08:26.689724 2839 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260429 14:08:26.689860 2839 mini_ranger.cc:66] Stopping Ranger...
I20260429 14:08:27.035552 2839 mini_ranger.cc:68] Stopped Ranger
2026-04-29 14:08:27.035 UTC [15020] LOG: received smart shutdown request
2026-04-29 14:08:27.039 UTC [15020] LOG: background worker "logical replication launcher" (PID 15028) exited with exit code 1
2026-04-29 14:08:27.040 UTC [15023] LOG: shutting down
2026-04-29 14:08:27.043 UTC [15023] LOG: checkpoint starting: shutdown immediate
2026-04-29 14:08:28.303 UTC [15023] LOG: checkpoint complete: wrote 2411 buffers (14.7%); 0 WAL file(s) added, 0 removed, 1 recycled; write=0.039 s, sync=1.213 s, total=1.264 s; sync files=1108, longest=0.007 s, average=0.002 s; distance=12073 kB, estimate=12073 kB; lsn=0/2044CD8, redo lsn=0/2044CD8
2026-04-29 14:08:28.312 UTC [15020] LOG: database system is shut down
2026-04-29T14:08:28Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (60630 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:08:28 dist-test-slave-8n77 krb5kdc[16849](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:08:28 dist-test-slave-8n77 krb5kdc[16849](info): set up 2 sockets
Apr 29 14:08:28 dist-test-slave-8n77 krb5kdc[16849](info): commencing operation
krb5kdc: starting...
W20260429 14:08:30.479368 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:08:30 dist-test-slave-8n77 krb5kdc[16849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:08:30Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:08:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:08:30.630084 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:43681
--webserver_interface=127.2.197.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:34041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:43681,127.2.197.253:38283,127.2.197.252:42177
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:08:30.708056 16865 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:08:30.708295 16865 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:08:30.708334 16865 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:08:30.709842 16865 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:08:30.709930 16865 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:08:30.709954 16865 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:08:30.709975 16865 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:08:30.710006 16865 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:08:30.711738 16865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:34041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:43681,127.2.197.253:38283,127.2.197.252:42177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:43681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:08:30.712049 16865 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:08:30.712249 16865 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:08:30.715335 16871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:08:30.715430 16865 server_base.cc:1061] running on GCE node
W20260429 14:08:30.715528 16870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:30.715446 16873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:08:30.715942 16865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:08:30.716161 16865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:08:30.717314 16865 hybrid_clock.cc:648] HybridClock initialized: now 1777471710717186 us; error 144 us; skew 500 ppm
Apr 29 14:08:30 dist-test-slave-8n77 krb5kdc[16849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471710, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:08:30.719707 16865 init.cc:377] Logged in from keytab as kudu/127.2.197.254@KRBTEST.COM (short username kudu)
I20260429 14:08:30.720297 16865 webserver.cc:492] Webserver started at http://127.2.197.254:34161/ using document root <none> and password file <none>
I20260429 14:08:30.720455 16865 fs_manager.cc:362] Metadata directory not provided
I20260429 14:08:30.720491 16865 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:08:30.720562 16865 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:08:30.721477 16865 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "910fb37ad87a40c7bcbc66109e6ac484"
format_stamp: "Formatted at 2026-04-29 14:08:30 on dist-test-slave-8n77"
server_key: "330d91a42f23abb4235dcfaa215203ca"
server_key_iv: "972e46e87fa13a5aa445807f1e220b87"
server_key_version: "encryptionkey@0"
I20260429 14:08:30.721851 16865 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "910fb37ad87a40c7bcbc66109e6ac484"
format_stamp: "Formatted at 2026-04-29 14:08:30 on dist-test-slave-8n77"
server_key: "330d91a42f23abb4235dcfaa215203ca"
server_key_iv: "972e46e87fa13a5aa445807f1e220b87"
server_key_version: "encryptionkey@0"
I20260429 14:08:30.723250 16865 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260429 14:08:30.723994 16880 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:30.724184 16865 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:08:30.724265 16865 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "910fb37ad87a40c7bcbc66109e6ac484"
format_stamp: "Formatted at 2026-04-29 14:08:30 on dist-test-slave-8n77"
server_key: "330d91a42f23abb4235dcfaa215203ca"
server_key_iv: "972e46e87fa13a5aa445807f1e220b87"
server_key_version: "encryptionkey@0"
I20260429 14:08:30.724349 16865 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:08:30.759068 16865 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:08:30.761787 16865 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:08:30.761981 16865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:08:30.767326 16865 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:43681
I20260429 14:08:30.767371 16933 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:43681 every 8 connection(s)
I20260429 14:08:30.767748 16865 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:08:30.768325 16934 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:08:30.770180 16934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43681 } has no permanent_uuid. Determining permanent_uuid...
Apr 29 14:08:30 dist-test-slave-8n77 krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471710, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:08:30.775974 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 16865
I20260429 14:08:30.776088 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:08:30.776274 2839 external_mini_cluster.cc:1468] Setting key 1927bb8e0509819e0977e5800b7829e0
I20260429 14:08:30.780170 16936 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:30.770940 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:43681 (local address 127.0.0.1:57022)
0429 14:08:30.771129 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:30.771171 (+ 42us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:30.771526 (+ 355us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:30.771835 (+ 309us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:30.771838 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:30.772148 (+ 310us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:08:30.772392 (+ 244us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:30.772394 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:30.773468 (+ 1074us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:30.773469 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:30.773860 (+ 391us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:30.773862 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:30.773985 (+ 123us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:30.774647 (+ 662us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:08:30.774657 (+ 10us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:08:30.776233 (+ 1576us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:08:30.778595 (+ 2362us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:30.778601 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:30.778603 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:30.778792 (+ 189us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:30.779011 (+ 219us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:30.779011 (+ 0us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:30.779012 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:30.779056 (+ 44us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:30.779485 (+ 429us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:08:30.779490 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:08:30.779601 (+ 111us) client_negotiation.cc:770] Sending connection context
0429 14:08:30.779725 (+ 124us) client_negotiation.cc:241] Negotiation successful
0429 14:08:30.780067 (+ 342us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":116,"thread_start_us":119,"threads_started":1}
I20260429 14:08:30.780182 16937 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:30.771020 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57022 (local address 127.2.197.254:43681)
0429 14:08:30.771169 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:30.771179 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:30.771254 (+ 75us) server_negotiation.cc:408] Connection header received
0429 14:08:30.771672 (+ 418us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:30.771674 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:30.771691 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:30.771776 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:30.772561 (+ 785us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:30.773352 (+ 791us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:30.773968 (+ 616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:30.774121 (+ 153us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:30.776377 (+ 2256us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:30.776387 (+ 10us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:30.776387 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:30.776395 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:30.778504 (+ 2109us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:30.778887 (+ 383us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:30.778890 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:30.778891 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:30.778944 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:30.779117 (+ 173us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:30.779118 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:30.779118 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:30.779392 (+ 274us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:30.779485 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:30.779782 (+ 297us) server_negotiation.cc:300] Negotiation successful
0429 14:08:30.779954 (+ 172us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260429 14:08:30.781181 16934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.197.253" port: 38283 } has no permanent_uuid. Determining permanent_uuid...
I20260429 14:08:30.781689 16936 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:30.781527 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.253:38283 (local address 127.0.0.1:44228)
0429 14:08:30.781591 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:30.781652 (+ 61us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.253:38283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260429 14:08:30.781862 16884 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.197.253:38283: connect: Connection refused (error 111)
I20260429 14:08:30.782431 16936 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:30.782270 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.253:38283 (local address 127.0.0.1:44244)
0429 14:08:30.782355 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:30.782393 (+ 38us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.253:38283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260429 14:08:30.782639 16934 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.197.253:38283: Network error: Client connection negotiation failed: client connection to 127.2.197.253:38283: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.2.197.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.253@KRBTEST.COM" created.
I20260429 14:08:30.831862 16934 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.197.253" port: 38283 } attempt: 1
I20260429 14:08:30.832350 16936 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:30.832198 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.253:38283 (local address 127.0.0.1:44250)
0429 14:08:30.832265 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:30.832315 (+ 50us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.253:38283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260429 14:08:30.832741 16936 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:30.832639 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.253:38283 (local address 127.0.0.1:44260)
0429 14:08:30.832695 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:30.832721 (+ 26us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.253:38283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260429 14:08:30.832849 16934 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.197.253:38283: Network error: Client connection negotiation failed: client connection to 127.2.197.253:38283: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:08:30.834156 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.253:38283
--webserver_interface=127.2.197.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:34041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:43681,127.2.197.253:38283,127.2.197.252:42177
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260429 14:08:30.901062 16934 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.197.253" port: 38283 } attempt: 2
I20260429 14:08:30.901636 16936 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:30.901473 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.253:38283 (local address 127.0.0.1:44268)
0429 14:08:30.901545 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:30.901596 (+ 51us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.253:38283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260429 14:08:30.902068 16936 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:30.901955 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.253:38283 (local address 127.0.0.1:44284)
0429 14:08:30.902008 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:30.902040 (+ 32us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.253:38283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260429 14:08:30.902199 16934 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.197.253:38283: Network error: Client connection negotiation failed: client connection to 127.2.197.253:38283: connect: Connection refused (error 111)
W20260429 14:08:30.914662 16943 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:08:30.914844 16943 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:08:30.914868 16943 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:08:30.916456 16943 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:08:30.916535 16943 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:08:30.916553 16943 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:08:30.916564 16943 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:08:30.916576 16943 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:08:30.918473 16943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:34041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:43681,127.2.197.253:38283,127.2.197.252:42177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.253:38283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.197.253
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:08:30.918737 16943 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:08:30.918941 16943 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:08:30.921429 16948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:30.921514 16949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:30.921593 16951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:08:30.921741 16943 server_base.cc:1061] running on GCE node
I20260429 14:08:30.921968 16943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:08:30.922221 16943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:08:30.923501 16943 hybrid_clock.cc:648] HybridClock initialized: now 1777471710923481 us; error 29 us; skew 500 ppm
Apr 29 14:08:30 dist-test-slave-8n77 krb5kdc[16849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471710, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:08:30.925848 16943 init.cc:377] Logged in from keytab as kudu/127.2.197.253@KRBTEST.COM (short username kudu)
I20260429 14:08:30.926617 16943 webserver.cc:492] Webserver started at http://127.2.197.253:41999/ using document root <none> and password file <none>
I20260429 14:08:30.926831 16943 fs_manager.cc:362] Metadata directory not provided
I20260429 14:08:30.926894 16943 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:08:30.927007 16943 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:08:30.928050 16943 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-1/data/instance:
uuid: "30991ca8f9ea4f43af9603b262823a13"
format_stamp: "Formatted at 2026-04-29 14:08:30 on dist-test-slave-8n77"
server_key: "7eee6a5eecf62504e29b17cc6c3992e2"
server_key_iv: "1aea4a0ec870d75628ee9eeff2f3830a"
server_key_version: "encryptionkey@0"
I20260429 14:08:30.928512 16943 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-1/wal/instance:
uuid: "30991ca8f9ea4f43af9603b262823a13"
format_stamp: "Formatted at 2026-04-29 14:08:30 on dist-test-slave-8n77"
server_key: "7eee6a5eecf62504e29b17cc6c3992e2"
server_key_iv: "1aea4a0ec870d75628ee9eeff2f3830a"
server_key_version: "encryptionkey@0"
I20260429 14:08:30.929848 16943 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260429 14:08:30.930773 16958 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:30.930986 16943 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260429 14:08:30.931088 16943 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-1/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-1/wal
uuid: "30991ca8f9ea4f43af9603b262823a13"
format_stamp: "Formatted at 2026-04-29 14:08:30 on dist-test-slave-8n77"
server_key: "7eee6a5eecf62504e29b17cc6c3992e2"
server_key_iv: "1aea4a0ec870d75628ee9eeff2f3830a"
server_key_version: "encryptionkey@0"
I20260429 14:08:30.931175 16943 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:08:30.957911 16943 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:08:30.960166 16943 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:08:30.960351 16943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:08:30.964519 16943 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.253:38283
I20260429 14:08:30.964583 17010 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.253:38283 every 8 connection(s)
I20260429 14:08:30.964900 16943 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-1/data/info.pb
I20260429 14:08:30.965479 17011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:08:30.967423 17011 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43681 } has no permanent_uuid. Determining permanent_uuid...
I20260429 14:08:30.969158 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 16943
I20260429 14:08:30.969254 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-1/wal/instance
I20260429 14:08:30.969455 2839 external_mini_cluster.cc:1468] Setting key 54c44074c6dc0f2ec8b13de64613b8c8
Apr 29 14:08:30 dist-test-slave-8n77 krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471710, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.253@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:08:30.976325 16937 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:30.968147 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57024 (local address 127.2.197.254:43681)
0429 14:08:30.968214 (+ 67us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:30.968217 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:30.968386 (+ 169us) server_negotiation.cc:408] Connection header received
0429 14:08:30.968788 (+ 402us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:30.968789 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:30.968803 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:30.968869 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:30.969804 (+ 935us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:30.970256 (+ 452us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:30.970861 (+ 605us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:30.971047 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:30.973357 (+ 2310us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:30.973363 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:30.973364 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:30.973376 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:30.974932 (+ 1556us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:30.975428 (+ 496us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:30.975429 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:30.975429 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:30.975451 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:30.975687 (+ 236us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:30.975688 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:30.975689 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:30.975797 (+ 108us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:30.975874 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:30.976134 (+ 260us) server_negotiation.cc:300] Negotiation successful
0429 14:08:30.976230 (+ 96us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260429 14:08:30.976423 17013 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:30.968096 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:43681 (local address 127.0.0.1:57024)
0429 14:08:30.968273 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:30.968315 (+ 42us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:30.968683 (+ 368us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:30.968960 (+ 277us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:30.968964 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:30.969326 (+ 362us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:08:30.969692 (+ 366us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:30.969697 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:30.970386 (+ 689us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:30.970387 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:30.970770 (+ 383us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:30.970772 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:30.970925 (+ 153us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:30.971506 (+ 581us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:08:30.971515 (+ 9us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:08:30.973238 (+ 1723us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:08:30.975079 (+ 1841us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:30.975086 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:30.975089 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:30.975295 (+ 206us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:30.975521 (+ 226us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:30.975521 (+ 0us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:30.975522 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:30.975594 (+ 72us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:30.975864 (+ 270us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:08:30.975869 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:08:30.975994 (+ 125us) client_negotiation.cc:770] Sending connection context
0429 14:08:30.976108 (+ 114us) client_negotiation.cc:241] Negotiation successful
0429 14:08:30.976250 (+ 142us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for kudu/127.2.197.252@KRBTEST.COM; defaulting to no policy
I20260429 14:08:30.977044 17011 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.197.253" port: 38283 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.252@KRBTEST.COM" created.
Apr 29 14:08:30 dist-test-slave-8n77 krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471710, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.253@KRBTEST.COM for kudu/127.2.197.253@KRBTEST.COM
I20260429 14:08:30.985044 17013 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:30.977376 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.253:38283 (local address 127.0.0.1:44294)
0429 14:08:30.977485 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:30.977498 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:30.977564 (+ 66us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:30.977914 (+ 350us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:30.977915 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:30.978045 (+ 130us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:08:30.978263 (+ 218us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:30.978265 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:30.979241 (+ 976us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:30.979243 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:30.979637 (+ 394us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:30.979639 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:30.979704 (+ 65us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:30.980391 (+ 687us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:08:30.980398 (+ 7us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:08:30.981880 (+ 1482us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:08:30.983806 (+ 1926us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:30.983808 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:30.983808 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:30.984013 (+ 205us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:30.984241 (+ 228us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:30.984242 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:30.984243 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:30.984281 (+ 38us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:30.984569 (+ 288us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:08:30.984570 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:08:30.984627 (+ 57us) client_negotiation.cc:770] Sending connection context
0429 14:08:30.984711 (+ 84us) client_negotiation.cc:241] Negotiation successful
0429 14:08:30.984839 (+ 128us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260429 14:08:30.985046 17015 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:30.977504 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44294 (local address 127.2.197.253:38283)
0429 14:08:30.977753 (+ 249us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:30.977760 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:30.977778 (+ 18us) server_negotiation.cc:408] Connection header received
0429 14:08:30.977797 (+ 19us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:30.977799 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:30.977811 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:30.977883 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:30.978396 (+ 513us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:30.979149 (+ 753us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:30.979749 (+ 600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:30.979985 (+ 236us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:30.981982 (+ 1997us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:30.981991 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:30.981992 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:30.981998 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:30.983711 (+ 1713us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:30.984125 (+ 414us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:30.984128 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:30.984128 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:30.984160 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:30.984376 (+ 216us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:30.984377 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:30.984378 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:30.984494 (+ 116us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:30.984577 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:30.984744 (+ 167us) server_negotiation.cc:300] Negotiation successful
0429 14:08:30.984922 (+ 178us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"mutex_wait_us":89,"server-negotiator.queue_time_us":207,"thread_start_us":153,"threads_started":1}
I20260429 14:08:30.985777 17011 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42177 } has no permanent_uuid. Determining permanent_uuid...
I20260429 14:08:30.986186 17013 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:30.986044 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:42177 (local address 127.0.0.1:46486)
0429 14:08:30.986096 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:30.986152 (+ 56us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.252:42177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260429 14:08:30.986395 16960 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.197.252:42177: connect: Connection refused (error 111)
I20260429 14:08:30.986873 17013 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:30.986770 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:42177 (local address 127.0.0.1:46496)
0429 14:08:30.986822 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:30.986850 (+ 28us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.252:42177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260429 14:08:30.986999 17011 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.197.252:42177: Network error: Client connection negotiation failed: client connection to 127.2.197.252:42177: connect: Connection refused (error 111)
I20260429 14:08:30.993407 16934 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.197.253" port: 38283 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Apr 29 14:08:30 dist-test-slave-8n77 krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471710, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for kudu/127.2.197.253@KRBTEST.COM
I20260429 14:08:31.000702 17015 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:30.993933 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44298 (local address 127.2.197.253:38283)
0429 14:08:30.994002 (+ 69us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:30.994032 (+ 30us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:30.994041 (+ 9us) server_negotiation.cc:408] Connection header received
0429 14:08:30.994170 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:30.994171 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:30.994188 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:30.994277 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:30.994867 (+ 590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:30.995382 (+ 515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:30.995955 (+ 573us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:30.996207 (+ 252us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:30.998107 (+ 1900us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:30.998111 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:30.998111 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:30.998119 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:30.999593 (+ 1474us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:30.999969 (+ 376us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:30.999970 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:30.999970 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:31.000003 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:31.000154 (+ 151us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:31.000155 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:31.000156 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:31.000258 (+ 102us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:31.000332 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:31.000510 (+ 178us) server_negotiation.cc:300] Negotiation successful
0429 14:08:31.000598 (+ 88us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260429 14:08:31.000702 16936 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:30.993819 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.253:38283 (local address 127.0.0.1:44298)
0429 14:08:30.993886 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:30.993898 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:30.993970 (+ 72us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:30.994427 (+ 457us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:30.994428 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:30.994572 (+ 144us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:08:30.994746 (+ 174us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:30.994748 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:30.995499 (+ 751us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:30.995500 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:30.995865 (+ 365us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:30.995867 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:30.995931 (+ 64us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:30.996522 (+ 591us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:08:30.996527 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:08:30.997987 (+ 1460us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:08:30.999712 (+ 1725us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:30.999713 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:30.999714 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:30.999874 (+ 160us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:31.000068 (+ 194us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:31.000069 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:31.000069 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:31.000099 (+ 30us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:31.000321 (+ 222us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:08:31.000322 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:08:31.000377 (+ 55us) client_negotiation.cc:770] Sending connection context
0429 14:08:31.000481 (+ 104us) client_negotiation.cc:241] Negotiation successful
0429 14:08:31.000597 (+ 116us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260429 14:08:31.001070 16934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42177 } has no permanent_uuid. Determining permanent_uuid...
I20260429 14:08:31.001447 16936 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:31.001335 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:42177 (local address 127.0.0.1:46508)
0429 14:08:31.001392 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:31.001421 (+ 29us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.252:42177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260429 14:08:31.002053 16936 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:31.001952 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:42177 (local address 127.0.0.1:46514)
0429 14:08:31.001999 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:31.002031 (+ 32us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.252:42177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260429 14:08:31.002161 16934 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.197.252:42177: Network error: Client connection negotiation failed: client connection to 127.2.197.252:42177: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.2.197.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:08:31.026168 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.252:42177
--webserver_interface=127.2.197.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:34041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:43681,127.2.197.253:38283,127.2.197.252:42177
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260429 14:08:31.033331 16934 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42177 } attempt: 1
I20260429 14:08:31.033890 16936 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:31.033717 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:42177 (local address 127.0.0.1:46528)
0429 14:08:31.033790 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:31.033841 (+ 51us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.252:42177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260429 14:08:31.034375 16936 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:31.034233 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:42177 (local address 127.0.0.1:46530)
0429 14:08:31.034301 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:31.034334 (+ 33us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.252:42177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260429 14:08:31.034508 16934 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.197.252:42177: Network error: Client connection negotiation failed: client connection to 127.2.197.252:42177: connect: Connection refused (error 111)
I20260429 14:08:31.036197 17011 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42177 } attempt: 1
I20260429 14:08:31.036631 17013 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:31.036497 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:42177 (local address 127.0.0.1:46538)
0429 14:08:31.036558 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:31.036595 (+ 37us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.252:42177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260429 14:08:31.037002 17013 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:31.036897 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:42177 (local address 127.0.0.1:46544)
0429 14:08:31.036946 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:31.036978 (+ 32us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.252:42177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260429 14:08:31.037128 17011 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.197.252:42177: Network error: Client connection negotiation failed: client connection to 127.2.197.252:42177: connect: Connection refused (error 111)
W20260429 14:08:31.105258 17020 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
I20260429 14:08:31.105319 17011 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42177 } attempt: 2
W20260429 14:08:31.105495 17020 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:08:31.105535 17020 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260429 14:08:31.105896 17013 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:31.105729 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:42177 (local address 127.0.0.1:46556)
0429 14:08:31.105812 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:31.105859 (+ 47us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.252:42177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260429 14:08:31.106366 17013 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:31.106242 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:42177 (local address 127.0.0.1:46566)
0429 14:08:31.106313 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:31.106342 (+ 29us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.252:42177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260429 14:08:31.106537 17011 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.197.252:42177: Network error: Client connection negotiation failed: client connection to 127.2.197.252:42177: connect: Connection refused (error 111)
W20260429 14:08:31.107486 17020 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:08:31.107529 17020 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:08:31.107544 17020 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:08:31.107558 17020 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:08:31.107569 17020 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:08:31.109378 17020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:34041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:43681,127.2.197.253:38283,127.2.197.252:42177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.252:42177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.197.252
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:08:31.109647 17020 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:08:31.109714 16934 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42177 } attempt: 2
I20260429 14:08:31.109838 17020 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260429 14:08:31.110142 16936 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:31.110007 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:42177 (local address 127.0.0.1:46574)
0429 14:08:31.110067 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:31.110106 (+ 39us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.252:42177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260429 14:08:31.110524 16936 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:31.110420 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:42177 (local address 127.0.0.1:46586)
0429 14:08:31.110470 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:31.110500 (+ 30us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.252:42177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260429 14:08:31.110628 16934 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.197.252:42177: Network error: Client connection negotiation failed: client connection to 127.2.197.252:42177: connect: Connection refused (error 111)
W20260429 14:08:31.112592 17026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:31.112620 17025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:08:31.112721 17020 server_base.cc:1061] running on GCE node
W20260429 14:08:31.112592 17028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:08:31.113080 17020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:08:31.113291 17020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:08:31.114431 17020 hybrid_clock.cc:648] HybridClock initialized: now 1777471711114392 us; error 52 us; skew 500 ppm
Apr 29 14:08:31 dist-test-slave-8n77 krb5kdc[16849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471711, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:08:31.116842 17020 init.cc:377] Logged in from keytab as kudu/127.2.197.252@KRBTEST.COM (short username kudu)
I20260429 14:08:31.117566 17020 webserver.cc:492] Webserver started at http://127.2.197.252:43165/ using document root <none> and password file <none>
I20260429 14:08:31.117792 17020 fs_manager.cc:362] Metadata directory not provided
I20260429 14:08:31.117856 17020 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:08:31.117960 17020 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:08:31.118973 17020 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-2/data/instance:
uuid: "40359b022e1b4ec591c7972831f8ce16"
format_stamp: "Formatted at 2026-04-29 14:08:31 on dist-test-slave-8n77"
server_key: "20d757b177a0344be588599afe670261"
server_key_iv: "1f5f64492f6c6a4598c772edc373d330"
server_key_version: "encryptionkey@0"
I20260429 14:08:31.119304 17020 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-2/wal/instance:
uuid: "40359b022e1b4ec591c7972831f8ce16"
format_stamp: "Formatted at 2026-04-29 14:08:31 on dist-test-slave-8n77"
server_key: "20d757b177a0344be588599afe670261"
server_key_iv: "1f5f64492f6c6a4598c772edc373d330"
server_key_version: "encryptionkey@0"
I20260429 14:08:31.120613 17020 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260429 14:08:31.121377 17035 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:31.121606 17020 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:08:31.121690 17020 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-2/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-2/wal
uuid: "40359b022e1b4ec591c7972831f8ce16"
format_stamp: "Formatted at 2026-04-29 14:08:31 on dist-test-slave-8n77"
server_key: "20d757b177a0344be588599afe670261"
server_key_iv: "1f5f64492f6c6a4598c772edc373d330"
server_key_version: "encryptionkey@0"
I20260429 14:08:31.121768 17020 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:08:31.163498 17020 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:08:31.165540 17020 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:08:31.165715 17020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:08:31.170254 17020 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.252:42177
I20260429 14:08:31.170358 17087 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.252:42177 every 8 connection(s)
I20260429 14:08:31.170734 17020 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-2/data/info.pb
I20260429 14:08:31.171309 17088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:08:31.171877 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 17020
I20260429 14:08:31.171985 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777471413505459-2839-0/minicluster-data/master-2/wal/instance
I20260429 14:08:31.172151 2839 external_mini_cluster.cc:1468] Setting key 0afd7d9b5d8a1e61cfa273b0d44d284b
I20260429 14:08:31.173858 17088 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43681 } has no permanent_uuid. Determining permanent_uuid...
Apr 29 14:08:31 dist-test-slave-8n77 krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471711, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.252@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
Apr 29 14:08:31 dist-test-slave-8n77 krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.252@KRBTEST.COM
I20260429 14:08:31.181944 16937 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:31.174541 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57038 (local address 127.2.197.254:43681)
0429 14:08:31.174612 (+ 71us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:31.174615 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:31.174772 (+ 157us) server_negotiation.cc:408] Connection header received
0429 14:08:31.174834 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:31.174835 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:31.174847 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:31.174912 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:31.175506 (+ 594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:31.175925 (+ 419us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:31.176558 (+ 633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:31.176764 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:31.178740 (+ 1976us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:31.178745 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:31.178745 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:31.178754 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:31.180365 (+ 1611us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:31.181030 (+ 665us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:31.181031 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:31.181031 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:31.181052 (+ 21us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:31.181251 (+ 199us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:31.181251 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:31.181251 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:31.181341 (+ 90us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:31.181389 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:31.181736 (+ 347us) server_negotiation.cc:300] Negotiation successful
0429 14:08:31.181826 (+ 90us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260429 14:08:31.181984 17092 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:31.174470 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:43681 (local address 127.0.0.1:57038)
0429 14:08:31.174676 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:31.174701 (+ 25us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:31.174779 (+ 78us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:31.174923 (+ 144us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:31.174925 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:31.175204 (+ 279us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:08:31.175385 (+ 181us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:31.175386 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:31.176050 (+ 664us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:31.176052 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:31.176474 (+ 422us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:31.176476 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:31.176582 (+ 106us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:31.176994 (+ 412us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:08:31.177004 (+ 10us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:08:31.178644 (+ 1640us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:08:31.180494 (+ 1850us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:31.180499 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:31.180502 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:31.180945 (+ 443us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:31.181117 (+ 172us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:31.181118 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:31.181119 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:31.181191 (+ 72us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:31.181406 (+ 215us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:08:31.181410 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:08:31.181557 (+ 147us) client_negotiation.cc:770] Sending connection context
0429 14:08:31.181687 (+ 130us) client_negotiation.cc:241] Negotiation successful
0429 14:08:31.181824 (+ 137us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":110,"threads_started":1}
I20260429 14:08:31.182614 17088 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.197.253" port: 38283 } has no permanent_uuid. Determining permanent_uuid...
I20260429 14:08:31.183008 17091 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:31.172731 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:46602 (local address 127.2.197.252:42177)
0429 14:08:31.172903 (+ 172us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:31.172915 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:31.172932 (+ 17us) server_negotiation.cc:408] Connection header received
0429 14:08:31.173285 (+ 353us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:31.173291 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:31.173328 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:31.173505 (+ 177us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:31.174367 (+ 862us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:31.175338 (+ 971us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:31.175991 (+ 653us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:31.176254 (+ 263us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:31.179434 (+ 3180us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:31.179445 (+ 11us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:31.179445 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:31.179458 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:31.181654 (+ 2196us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:31.182014 (+ 360us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:31.182017 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:31.182017 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:31.182041 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:31.182299 (+ 258us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:31.182300 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:31.182300 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:31.182450 (+ 150us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:31.182614 (+ 164us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:31.182735 (+ 121us) server_negotiation.cc:300] Negotiation successful
0429 14:08:31.182826 (+ 91us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":83,"threads_started":1}
Apr 29 14:08:31 dist-test-slave-8n77 krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471711, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.252@KRBTEST.COM for kudu/127.2.197.253@KRBTEST.COM
Apr 29 14:08:31 dist-test-slave-8n77 krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:08:31.189751 17015 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:31.182984 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44302 (local address 127.2.197.253:38283)
0429 14:08:31.183033 (+ 49us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:31.183035 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:31.183100 (+ 65us) server_negotiation.cc:408] Connection header received
0429 14:08:31.183205 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:31.183206 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:31.183217 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:31.183245 (+ 28us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:31.183799 (+ 554us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:31.184489 (+ 690us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:31.185019 (+ 530us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:31.185207 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:31.187097 (+ 1890us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:31.187102 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:31.187102 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:31.187110 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:31.188635 (+ 1525us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:31.189019 (+ 384us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:31.189020 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:31.189021 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:31.189047 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:31.189248 (+ 201us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:31.189248 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:31.189249 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:31.189343 (+ 94us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:31.189398 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:31.189602 (+ 204us) server_negotiation.cc:300] Negotiation successful
0429 14:08:31.189684 (+ 82us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260429 14:08:31.189805 17092 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:31.182923 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.253:38283 (local address 127.0.0.1:44302)
0429 14:08:31.183033 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:31.183043 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:31.183132 (+ 89us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:31.183310 (+ 178us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:31.183311 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:31.183493 (+ 182us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:08:31.183694 (+ 201us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:31.183697 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:31.184585 (+ 888us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:31.184585 (+ 0us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:31.184930 (+ 345us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:31.184932 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:31.184983 (+ 51us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:31.185431 (+ 448us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:08:31.185435 (+ 4us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:08:31.186995 (+ 1560us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:08:31.188739 (+ 1744us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:31.188741 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:31.188741 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:31.188938 (+ 197us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:31.189137 (+ 199us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:31.189138 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:31.189138 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:31.189174 (+ 36us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:31.189444 (+ 270us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:08:31.189445 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:08:31.189496 (+ 51us) client_negotiation.cc:770] Sending connection context
0429 14:08:31.189575 (+ 79us) client_negotiation.cc:241] Negotiation successful
0429 14:08:31.189682 (+ 107us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260429 14:08:31.190382 17088 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42177 } has no permanent_uuid. Determining permanent_uuid...
I20260429 14:08:31.191239 16937 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:31.183765 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57042 (local address 127.2.197.254:43681)
0429 14:08:31.183807 (+ 42us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:31.183809 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:31.183815 (+ 6us) server_negotiation.cc:408] Connection header received
0429 14:08:31.183831 (+ 16us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:31.183832 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:31.183839 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:31.183884 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:31.184585 (+ 701us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:31.185079 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:31.185684 (+ 605us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:31.185877 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:31.188134 (+ 2257us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:31.188138 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:31.188138 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:31.188146 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:31.189926 (+ 1780us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:31.190361 (+ 435us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:31.190362 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:31.190363 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:31.190390 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:31.190693 (+ 303us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:31.190694 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:31.190695 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:31.190832 (+ 137us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:31.191011 (+ 179us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:31.191048 (+ 37us) server_negotiation.cc:300] Negotiation successful
0429 14:08:31.191132 (+ 84us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Apr 29 14:08:31 dist-test-slave-8n77 krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471711, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.252@KRBTEST.COM for kudu/127.2.197.252@KRBTEST.COM
Apr 29 14:08:31 dist-test-slave-8n77 krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.253@KRBTEST.COM
I20260429 14:08:31.197448 17092 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:31.190763 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:42177 (local address 127.0.0.1:46614)
0429 14:08:31.190854 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:31.190865 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:31.190911 (+ 46us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:31.191029 (+ 118us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:31.191030 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:31.191167 (+ 137us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:08:31.191309 (+ 142us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:31.191312 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:31.192294 (+ 982us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:31.192295 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:31.192714 (+ 419us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:31.192716 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:31.192773 (+ 57us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:31.193208 (+ 435us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:08:31.193212 (+ 4us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:08:31.194525 (+ 1313us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:08:31.196310 (+ 1785us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:31.196311 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:31.196312 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:31.196503 (+ 191us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:31.196738 (+ 235us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:31.196740 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:31.196740 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:31.196775 (+ 35us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:31.197097 (+ 322us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:08:31.197097 (+ 0us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:08:31.197137 (+ 40us) client_negotiation.cc:770] Sending connection context
0429 14:08:31.197195 (+ 58us) client_negotiation.cc:241] Negotiation successful
0429 14:08:31.197317 (+ 122us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260429 14:08:31.197551 17091 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:31.190820 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:46614 (local address 127.2.197.252:42177)
0429 14:08:31.190857 (+ 37us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:31.190860 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:31.190896 (+ 36us) server_negotiation.cc:408] Connection header received
0429 14:08:31.190964 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:31.190965 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:31.190974 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:31.191027 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:31.191384 (+ 357us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:31.192209 (+ 825us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:31.192798 (+ 589us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:31.192986 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:31.194624 (+ 1638us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:31.194628 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:31.194628 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:31.194634 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:31.196217 (+ 1583us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:31.196577 (+ 360us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:31.196578 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:31.196579 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:31.196603 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:31.196877 (+ 274us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:31.196878 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:31.196879 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:31.197020 (+ 141us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:31.197099 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:31.197244 (+ 145us) server_negotiation.cc:300] Negotiation successful
0429 14:08:31.197373 (+ 129us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":18}
I20260429 14:08:31.197693 17011 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42177 } attempt: 3
I20260429 14:08:31.199397 17015 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:31.192127 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44306 (local address 127.2.197.253:38283)
0429 14:08:31.192210 (+ 83us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:31.192211 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:31.192220 (+ 9us) server_negotiation.cc:408] Connection header received
0429 14:08:31.192254 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:31.192256 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:31.192266 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:31.192328 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:31.192944 (+ 616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:31.193370 (+ 426us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:31.193848 (+ 478us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:31.194028 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:31.196469 (+ 2441us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:31.196472 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:31.196473 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:31.196480 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:31.198247 (+ 1767us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:31.198699 (+ 452us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:31.198701 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:31.198701 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:31.198725 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:31.198907 (+ 182us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:31.198907 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:31.198908 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:31.198998 (+ 90us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:31.199142 (+ 144us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:31.199204 (+ 62us) server_negotiation.cc:300] Negotiation successful
0429 14:08:31.199290 (+ 86us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260429 14:08:31.199510 17088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40359b022e1b4ec591c7972831f8ce16: Bootstrap starting.
I20260429 14:08:31.200502 17088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40359b022e1b4ec591c7972831f8ce16: Neither blocks nor log segments found. Creating new log.
I20260429 14:08:31.200796 17088 log.cc:826] T 00000000000000000000000000000000 P 40359b022e1b4ec591c7972831f8ce16: Log is configured to *not* fsync() on all Append() calls
I20260429 14:08:31.200834 2839 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
Apr 29 14:08:31 dist-test-slave-8n77 krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471710, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.253@KRBTEST.COM for kudu/127.2.197.252@KRBTEST.COM
I20260429 14:08:31.201666 17088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40359b022e1b4ec591c7972831f8ce16: No bootstrap required, opened a new log
I20260429 14:08:31.203557 17088 raft_consensus.cc:359] T 00000000000000000000000000000000 P 40359b022e1b4ec591c7972831f8ce16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fb37ad87a40c7bcbc66109e6ac484" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43681 } } peers { permanent_uuid: "30991ca8f9ea4f43af9603b262823a13" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 38283 } } peers { permanent_uuid: "40359b022e1b4ec591c7972831f8ce16" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42177 } }
I20260429 14:08:31.203719 17088 raft_consensus.cc:385] T 00000000000000000000000000000000 P 40359b022e1b4ec591c7972831f8ce16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:08:31.203756 17088 raft_consensus.cc:740] T 00000000000000000000000000000000 P 40359b022e1b4ec591c7972831f8ce16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40359b022e1b4ec591c7972831f8ce16, State: Initialized, Role: FOLLOWER
I20260429 14:08:31.203859 17088 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40359b022e1b4ec591c7972831f8ce16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fb37ad87a40c7bcbc66109e6ac484" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43681 } } peers { permanent_uuid: "30991ca8f9ea4f43af9603b262823a13" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 38283 } } peers { permanent_uuid: "40359b022e1b4ec591c7972831f8ce16" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42177 } }
I20260429 14:08:31.204193 17088 sys_catalog.cc:565] T 00000000000000000000000000000000 P 40359b022e1b4ec591c7972831f8ce16 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:08:31.204169 17095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40359b022e1b4ec591c7972831f8ce16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fb37ad87a40c7bcbc66109e6ac484" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43681 } } peers { permanent_uuid: "30991ca8f9ea4f43af9603b262823a13" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 38283 } } peers { permanent_uuid: "40359b022e1b4ec591c7972831f8ce16" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42177 } } }
I20260429 14:08:31.204283 17095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40359b022e1b4ec591c7972831f8ce16 [sys.catalog]: This master's current role is: FOLLOWER
I20260429 14:08:31.205133 17091 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:31.198054 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:46628 (local address 127.2.197.252:42177)
0429 14:08:31.198106 (+ 52us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:31.198108 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:31.198116 (+ 8us) server_negotiation.cc:408] Connection header received
0429 14:08:31.198135 (+ 19us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:31.198136 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:31.198151 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:31.198220 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:31.198769 (+ 549us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:31.199371 (+ 602us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:31.200269 (+ 898us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:31.202054 (+ 1785us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:31.202099 (+ 45us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:31.202101 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:31.202102 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:31.202107 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:31.203914 (+ 1807us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:31.204343 (+ 429us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:31.204344 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:31.204345 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:31.204369 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:31.204610 (+ 241us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:31.204611 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:31.204611 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:31.204730 (+ 119us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:31.204811 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:31.204960 (+ 149us) server_negotiation.cc:300] Negotiation successful
0429 14:08:31.205045 (+ 85us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260429 14:08:31.205135 17013 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:31.197967 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:42177 (local address 127.0.0.1:46628)
0429 14:08:31.198011 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:31.198023 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:31.198072 (+ 49us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:31.198226 (+ 154us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:31.198227 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:31.198466 (+ 239us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:08:31.198684 (+ 218us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:31.198687 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:31.199477 (+ 790us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:31.199478 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:31.200017 (+ 539us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:31.200020 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:31.200080 (+ 60us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:31.200488 (+ 408us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:08:31.200494 (+ 6us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:08:31.201927 (+ 1433us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:08:31.204031 (+ 2104us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:31.204033 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:31.204033 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:31.204240 (+ 207us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:31.204502 (+ 262us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:31.204503 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:31.204504 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:31.204543 (+ 39us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:31.204803 (+ 260us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:08:31.204804 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:08:31.204851 (+ 47us) client_negotiation.cc:770] Sending connection context
0429 14:08:31.204937 (+ 86us) client_negotiation.cc:241] Negotiation successful
0429 14:08:31.205045 (+ 108us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
W20260429 14:08:31.206426 17106 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 40359b022e1b4ec591c7972831f8ce16: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260429 14:08:31.206542 17106 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260429 14:08:31.206771 17011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30991ca8f9ea4f43af9603b262823a13: Bootstrap starting.
I20260429 14:08:31.207705 17011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30991ca8f9ea4f43af9603b262823a13: Neither blocks nor log segments found. Creating new log.
I20260429 14:08:31.208002 17011 log.cc:826] T 00000000000000000000000000000000 P 30991ca8f9ea4f43af9603b262823a13: Log is configured to *not* fsync() on all Append() calls
I20260429 14:08:31.208726 17011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30991ca8f9ea4f43af9603b262823a13: No bootstrap required, opened a new log
I20260429 14:08:31.209921 16934 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42177 } attempt: 3
I20260429 14:08:31.210631 17011 raft_consensus.cc:359] T 00000000000000000000000000000000 P 30991ca8f9ea4f43af9603b262823a13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fb37ad87a40c7bcbc66109e6ac484" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43681 } } peers { permanent_uuid: "30991ca8f9ea4f43af9603b262823a13" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 38283 } } peers { permanent_uuid: "40359b022e1b4ec591c7972831f8ce16" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42177 } }
I20260429 14:08:31.210747 17011 raft_consensus.cc:385] T 00000000000000000000000000000000 P 30991ca8f9ea4f43af9603b262823a13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:08:31.210778 17011 raft_consensus.cc:740] T 00000000000000000000000000000000 P 30991ca8f9ea4f43af9603b262823a13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30991ca8f9ea4f43af9603b262823a13, State: Initialized, Role: FOLLOWER
I20260429 14:08:31.210871 17011 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30991ca8f9ea4f43af9603b262823a13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fb37ad87a40c7bcbc66109e6ac484" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43681 } } peers { permanent_uuid: "30991ca8f9ea4f43af9603b262823a13" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 38283 } } peers { permanent_uuid: "40359b022e1b4ec591c7972831f8ce16" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42177 } }
I20260429 14:08:31.209095 17020 master_runner.cc:186] Error getting master registration for 127.2.197.254:43681: OK, instance_id { permanent_uuid: "910fb37ad87a40c7bcbc66109e6ac484" instance_seqno: 1777471710765875 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260429 14:08:31.211154 17011 sys_catalog.cc:565] T 00000000000000000000000000000000 P 30991ca8f9ea4f43af9603b262823a13 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:08:31.211194 17020 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260429 14:08:31.213423 17122 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 30991ca8f9ea4f43af9603b262823a13: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260429 14:08:31.213478 17122 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260429 14:08:31.213667 17110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30991ca8f9ea4f43af9603b262823a13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fb37ad87a40c7bcbc66109e6ac484" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43681 } } peers { permanent_uuid: "30991ca8f9ea4f43af9603b262823a13" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 38283 } } peers { permanent_uuid: "40359b022e1b4ec591c7972831f8ce16" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42177 } } }
I20260429 14:08:31.213742 17110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30991ca8f9ea4f43af9603b262823a13 [sys.catalog]: This master's current role is: FOLLOWER
Apr 29 14:08:31 dist-test-slave-8n77 krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471710, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for kudu/127.2.197.252@KRBTEST.COM
I20260429 14:08:31.213238 16943 master_runner.cc:186] Error getting master registration for 127.2.197.254:43681: OK, instance_id { permanent_uuid: "910fb37ad87a40c7bcbc66109e6ac484" instance_seqno: 1777471710765875 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260429 14:08:31.217254 16943 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260429 14:08:31.220019 16936 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:31.210176 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:42177 (local address 127.0.0.1:46642)
0429 14:08:31.210264 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:31.210304 (+ 40us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:31.210346 (+ 42us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:31.210610 (+ 264us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:31.210612 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:31.210810 (+ 198us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:08:31.211021 (+ 211us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:31.211023 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:31.211792 (+ 769us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:31.211794 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:31.212348 (+ 554us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:31.212351 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:31.212438 (+ 87us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:31.212977 (+ 539us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:08:31.212983 (+ 6us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:08:31.215582 (+ 2599us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:08:31.218883 (+ 3301us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:31.218885 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:31.218885 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:31.219115 (+ 230us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:31.219307 (+ 192us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:31.219308 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:31.219309 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:31.219346 (+ 37us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:31.219661 (+ 315us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:08:31.219662 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:08:31.219715 (+ 53us) client_negotiation.cc:770] Sending connection context
0429 14:08:31.219786 (+ 71us) client_negotiation.cc:241] Negotiation successful
0429 14:08:31.219907 (+ 121us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260429 14:08:31.220464 17091 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:31.210429 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:46642 (local address 127.2.197.252:42177)
0429 14:08:31.210469 (+ 40us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:31.210471 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:31.210478 (+ 7us) server_negotiation.cc:408] Connection header received
0429 14:08:31.210499 (+ 21us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:31.210500 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:31.210510 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:31.210555 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:31.211233 (+ 678us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:31.211690 (+ 457us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:31.212635 (+ 945us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:31.212838 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:31.215705 (+ 2867us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:31.215711 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:31.215711 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:31.215719 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:31.218753 (+ 3034us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:31.219206 (+ 453us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:31.219208 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:31.219208 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:31.219236 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:31.219427 (+ 191us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:31.219427 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:31.219428 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:31.219566 (+ 138us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:31.219632 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:31.220248 (+ 616us) server_negotiation.cc:300] Negotiation successful
0429 14:08:31.220343 (+ 95us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260429 14:08:31.221989 16934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 910fb37ad87a40c7bcbc66109e6ac484: Bootstrap starting.
Apr 29 14:08:31 dist-test-slave-8n77 krb5kdc[16849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260429 14:08:31.222995 16934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 910fb37ad87a40c7bcbc66109e6ac484: Neither blocks nor log segments found. Creating new log.
I20260429 14:08:31.223269 16934 log.cc:826] T 00000000000000000000000000000000 P 910fb37ad87a40c7bcbc66109e6ac484: Log is configured to *not* fsync() on all Append() calls
Apr 29 14:08:31 dist-test-slave-8n77 krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:08:31.235239 16934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 910fb37ad87a40c7bcbc66109e6ac484: No bootstrap required, opened a new log
Apr 29 14:08:31 dist-test-slave-8n77 krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.252@KRBTEST.COM
I20260429 14:08:31.237251 16934 raft_consensus.cc:359] T 00000000000000000000000000000000 P 910fb37ad87a40c7bcbc66109e6ac484 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fb37ad87a40c7bcbc66109e6ac484" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43681 } } peers { permanent_uuid: "30991ca8f9ea4f43af9603b262823a13" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 38283 } } peers { permanent_uuid: "40359b022e1b4ec591c7972831f8ce16" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42177 } }
I20260429 14:08:31.237397 16934 raft_consensus.cc:385] T 00000000000000000000000000000000 P 910fb37ad87a40c7bcbc66109e6ac484 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:08:31.237442 16934 raft_consensus.cc:740] T 00000000000000000000000000000000 P 910fb37ad87a40c7bcbc66109e6ac484 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 910fb37ad87a40c7bcbc66109e6ac484, State: Initialized, Role: FOLLOWER
I20260429 14:08:31.237545 16934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 910fb37ad87a40c7bcbc66109e6ac484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fb37ad87a40c7bcbc66109e6ac484" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43681 } } peers { permanent_uuid: "30991ca8f9ea4f43af9603b262823a13" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 38283 } } peers { permanent_uuid: "40359b022e1b4ec591c7972831f8ce16" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42177 } }
I20260429 14:08:31.237819 16934 sys_catalog.cc:565] T 00000000000000000000000000000000 P 910fb37ad87a40c7bcbc66109e6ac484 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:08:31.239289 17091 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:31.229532 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:46650 (local address 127.2.197.252:42177)
0429 14:08:31.229579 (+ 47us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:31.229582 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:31.229590 (+ 8us) server_negotiation.cc:408] Connection header received
0429 14:08:31.229613 (+ 23us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:31.229614 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:31.229625 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:31.229673 (+ 48us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:31.231224 (+ 1551us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:31.231915 (+ 691us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:31.232651 (+ 736us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:31.232848 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:31.236161 (+ 3313us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:31.236166 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:31.236166 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:31.236174 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:31.237933 (+ 1759us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:31.238326 (+ 393us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:31.238328 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:31.238328 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:31.238354 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:31.238540 (+ 186us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:31.238541 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:31.238541 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:31.238663 (+ 122us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:31.238760 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:31.239103 (+ 343us) server_negotiation.cc:300] Negotiation successful
0429 14:08:31.239190 (+ 87us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260429 14:08:31.240006 16937 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:31.229234 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57052 (local address 127.2.197.254:43681)
0429 14:08:31.229335 (+ 101us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:31.229337 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:31.229348 (+ 11us) server_negotiation.cc:408] Connection header received
0429 14:08:31.229377 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:31.229378 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:31.229386 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:31.229453 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:31.230486 (+ 1033us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:31.231221 (+ 735us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:31.231962 (+ 741us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:31.232132 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:31.236075 (+ 3943us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:31.236080 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:31.236081 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:31.236089 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:31.238527 (+ 2438us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:31.238877 (+ 350us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:31.238878 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:31.238878 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:31.238902 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:31.239150 (+ 248us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:31.239151 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:31.239151 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:31.239257 (+ 106us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:31.239323 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:31.239820 (+ 497us) server_negotiation.cc:300] Negotiation successful
0429 14:08:31.239906 (+ 86us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260429 14:08:31.241853 16865 master_runner.cc:263] No leader master found from master 910fb37ad87a40c7bcbc66109e6ac484
I20260429 14:08:31.241900 16865 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Apr 29 14:08:31 dist-test-slave-8n77 krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.253@KRBTEST.COM
I20260429 14:08:31.244872 17110 raft_consensus.cc:493] T 00000000000000000000000000000000 P 30991ca8f9ea4f43af9603b262823a13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:08:31.244959 17110 raft_consensus.cc:515] T 00000000000000000000000000000000 P 30991ca8f9ea4f43af9603b262823a13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fb37ad87a40c7bcbc66109e6ac484" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43681 } } peers { permanent_uuid: "30991ca8f9ea4f43af9603b262823a13" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 38283 } } peers { permanent_uuid: "40359b022e1b4ec591c7972831f8ce16" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42177 } }
I20260429 14:08:31.245184 17110 leader_election.cc:290] T 00000000000000000000000000000000 P 30991ca8f9ea4f43af9603b262823a13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 910fb37ad87a40c7bcbc66109e6ac484 (127.2.197.254:43681), 40359b022e1b4ec591c7972831f8ce16 (127.2.197.252:42177)
I20260429 14:08:31.245505 17062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "30991ca8f9ea4f43af9603b262823a13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40359b022e1b4ec591c7972831f8ce16" is_pre_election: true
I20260429 14:08:31.245509 16907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "30991ca8f9ea4f43af9603b262823a13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "910fb37ad87a40c7bcbc66109e6ac484" is_pre_election: true
I20260429 14:08:31.245658 16907 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 910fb37ad87a40c7bcbc66109e6ac484 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30991ca8f9ea4f43af9603b262823a13 in term 0.
I20260429 14:08:31.245693 17062 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 40359b022e1b4ec591c7972831f8ce16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30991ca8f9ea4f43af9603b262823a13 in term 0.
I20260429 14:08:31.245829 16960 leader_election.cc:304] T 00000000000000000000000000000000 P 30991ca8f9ea4f43af9603b262823a13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30991ca8f9ea4f43af9603b262823a13, 910fb37ad87a40c7bcbc66109e6ac484; no voters:
I20260429 14:08:31.245991 17110 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 30991ca8f9ea4f43af9603b262823a13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 14:08:31.246045 17110 raft_consensus.cc:493] T 00000000000000000000000000000000 P 30991ca8f9ea4f43af9603b262823a13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 14:08:31.246073 17110 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 30991ca8f9ea4f43af9603b262823a13 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:08:31.246737 17132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 910fb37ad87a40c7bcbc66109e6ac484 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fb37ad87a40c7bcbc66109e6ac484" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43681 } } peers { permanent_uuid: "30991ca8f9ea4f43af9603b262823a13" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 38283 } } peers { permanent_uuid: "40359b022e1b4ec591c7972831f8ce16" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42177 } } }
I20260429 14:08:31.246829 17132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 910fb37ad87a40c7bcbc66109e6ac484 [sys.catalog]: This master's current role is: FOLLOWER
W20260429 14:08:31.246852 17143 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 910fb37ad87a40c7bcbc66109e6ac484: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260429 14:08:31.246901 17143 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260429 14:08:31.247095 17110 raft_consensus.cc:515] T 00000000000000000000000000000000 P 30991ca8f9ea4f43af9603b262823a13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fb37ad87a40c7bcbc66109e6ac484" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43681 } } peers { permanent_uuid: "30991ca8f9ea4f43af9603b262823a13" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 38283 } } peers { permanent_uuid: "40359b022e1b4ec591c7972831f8ce16" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42177 } }
I20260429 14:08:31.247239 17110 leader_election.cc:290] T 00000000000000000000000000000000 P 30991ca8f9ea4f43af9603b262823a13 [CANDIDATE]: Term 1 election: Requested vote from peers 910fb37ad87a40c7bcbc66109e6ac484 (127.2.197.254:43681), 40359b022e1b4ec591c7972831f8ce16 (127.2.197.252:42177)
I20260429 14:08:31.247452 16907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "30991ca8f9ea4f43af9603b262823a13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "910fb37ad87a40c7bcbc66109e6ac484"
I20260429 14:08:31.247505 17015 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:31.229743 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44308 (local address 127.2.197.253:38283)
0429 14:08:31.229782 (+ 39us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:31.229784 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:31.233160 (+ 3376us) server_negotiation.cc:408] Connection header received
0429 14:08:31.234179 (+ 1019us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:31.234181 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:31.234197 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:31.234230 (+ 33us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:31.235032 (+ 802us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:31.235707 (+ 675us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:31.240464 (+ 4757us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:31.240644 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:31.243411 (+ 2767us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:31.243415 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:31.243416 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:31.243424 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:31.244813 (+ 1389us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:31.245184 (+ 371us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:31.245185 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:31.245185 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:31.245215 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:31.245445 (+ 230us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:31.245446 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:31.245447 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:31.245568 (+ 121us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:31.245647 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:31.245854 (+ 207us) server_negotiation.cc:300] Negotiation successful
0429 14:08:31.245943 (+ 89us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260429 14:08:31.247493 17062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "30991ca8f9ea4f43af9603b262823a13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40359b022e1b4ec591c7972831f8ce16"
I20260429 14:08:31.247529 16907 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 910fb37ad87a40c7bcbc66109e6ac484 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:08:31.247560 17062 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 40359b022e1b4ec591c7972831f8ce16 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:08:31.248528 16907 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 910fb37ad87a40c7bcbc66109e6ac484 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30991ca8f9ea4f43af9603b262823a13 in term 1.
I20260429 14:08:31.248528 17062 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 40359b022e1b4ec591c7972831f8ce16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30991ca8f9ea4f43af9603b262823a13 in term 1.
I20260429 14:08:31.248715 16960 leader_election.cc:304] T 00000000000000000000000000000000 P 30991ca8f9ea4f43af9603b262823a13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30991ca8f9ea4f43af9603b262823a13, 40359b022e1b4ec591c7972831f8ce16; no voters:
I20260429 14:08:31.248790 17110 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 30991ca8f9ea4f43af9603b262823a13 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:08:31.248926 17110 raft_consensus.cc:697] T 00000000000000000000000000000000 P 30991ca8f9ea4f43af9603b262823a13 [term 1 LEADER]: Becoming Leader. State: Replica: 30991ca8f9ea4f43af9603b262823a13, State: Running, Role: LEADER
I20260429 14:08:31.249006 17110 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30991ca8f9ea4f43af9603b262823a13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fb37ad87a40c7bcbc66109e6ac484" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43681 } } peers { permanent_uuid: "30991ca8f9ea4f43af9603b262823a13" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 38283 } } peers { permanent_uuid: "40359b022e1b4ec591c7972831f8ce16" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42177 } }
I20260429 14:08:31.249413 17110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30991ca8f9ea4f43af9603b262823a13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30991ca8f9ea4f43af9603b262823a13. Latest consensus state: current_term: 1 leader_uuid: "30991ca8f9ea4f43af9603b262823a13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fb37ad87a40c7bcbc66109e6ac484" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43681 } } peers { permanent_uuid: "30991ca8f9ea4f43af9603b262823a13" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 38283 } } peers { permanent_uuid: "40359b022e1b4ec591c7972831f8ce16" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42177 } } }
I20260429 14:08:31.249490 17110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30991ca8f9ea4f43af9603b262823a13 [sys.catalog]: This master's current role is: LEADER
I20260429 14:08:31.254524 17147 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:08:31.254840 17147 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:08:31.256476 17062 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 40359b022e1b4ec591c7972831f8ce16 [term 1 FOLLOWER]: Refusing update from remote peer 30991ca8f9ea4f43af9603b262823a13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:08:31.256578 16907 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 910fb37ad87a40c7bcbc66109e6ac484 [term 1 FOLLOWER]: Refusing update from remote peer 30991ca8f9ea4f43af9603b262823a13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:08:31.256767 17110 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 30991ca8f9ea4f43af9603b262823a13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40359b022e1b4ec591c7972831f8ce16" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:08:31.257522 17145 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 30991ca8f9ea4f43af9603b262823a13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910fb37ad87a40c7bcbc66109e6ac484" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:08:31.258219 17095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40359b022e1b4ec591c7972831f8ce16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30991ca8f9ea4f43af9603b262823a13. Latest consensus state: current_term: 1 leader_uuid: "30991ca8f9ea4f43af9603b262823a13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fb37ad87a40c7bcbc66109e6ac484" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43681 } } peers { permanent_uuid: "30991ca8f9ea4f43af9603b262823a13" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 38283 } } peers { permanent_uuid: "40359b022e1b4ec591c7972831f8ce16" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42177 } } }
I20260429 14:08:31.258311 17095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40359b022e1b4ec591c7972831f8ce16 [sys.catalog]: This master's current role is: FOLLOWER
I20260429 14:08:31.258737 17132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 910fb37ad87a40c7bcbc66109e6ac484 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30991ca8f9ea4f43af9603b262823a13. Latest consensus state: current_term: 1 leader_uuid: "30991ca8f9ea4f43af9603b262823a13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fb37ad87a40c7bcbc66109e6ac484" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43681 } } peers { permanent_uuid: "30991ca8f9ea4f43af9603b262823a13" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 38283 } } peers { permanent_uuid: "40359b022e1b4ec591c7972831f8ce16" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42177 } } }
I20260429 14:08:31.258816 17132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 910fb37ad87a40c7bcbc66109e6ac484 [sys.catalog]: This master's current role is: FOLLOWER
I20260429 14:08:31.260684 17145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30991ca8f9ea4f43af9603b262823a13 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "30991ca8f9ea4f43af9603b262823a13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fb37ad87a40c7bcbc66109e6ac484" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43681 } } peers { permanent_uuid: "30991ca8f9ea4f43af9603b262823a13" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 38283 } } peers { permanent_uuid: "40359b022e1b4ec591c7972831f8ce16" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42177 } } }
I20260429 14:08:31.260764 17145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30991ca8f9ea4f43af9603b262823a13 [sys.catalog]: This master's current role is: LEADER
I20260429 14:08:31.261314 17132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 910fb37ad87a40c7bcbc66109e6ac484 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "30991ca8f9ea4f43af9603b262823a13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fb37ad87a40c7bcbc66109e6ac484" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43681 } } peers { permanent_uuid: "30991ca8f9ea4f43af9603b262823a13" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 38283 } } peers { permanent_uuid: "40359b022e1b4ec591c7972831f8ce16" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42177 } } }
I20260429 14:08:31.261392 17132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 910fb37ad87a40c7bcbc66109e6ac484 [sys.catalog]: This master's current role is: FOLLOWER
I20260429 14:08:31.261667 17148 mvcc.cc:204] Tried to move back new op lower bound from 7280524129303773184 to 7280524129276645376. Current Snapshot: MvccSnapshot[applied={T|T < 7280524129303773184}]
I20260429 14:08:31.261893 17151 mvcc.cc:204] Tried to move back new op lower bound from 7280524129303773184 to 7280524129276645376. Current Snapshot: MvccSnapshot[applied={T|T < 7280524129303773184}]
I20260429 14:08:31.262528 17145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30991ca8f9ea4f43af9603b262823a13 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "30991ca8f9ea4f43af9603b262823a13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fb37ad87a40c7bcbc66109e6ac484" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43681 } } peers { permanent_uuid: "30991ca8f9ea4f43af9603b262823a13" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 38283 } } peers { permanent_uuid: "40359b022e1b4ec591c7972831f8ce16" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42177 } } }
I20260429 14:08:31.262605 17145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30991ca8f9ea4f43af9603b262823a13 [sys.catalog]: This master's current role is: LEADER
I20260429 14:08:31.262826 17147 catalog_manager.cc:1357] Generated new cluster ID: 7b1792c4461641a09bab4c6d34e54867
I20260429 14:08:31.262869 17147 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:08:31.263074 17095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40359b022e1b4ec591c7972831f8ce16 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "30991ca8f9ea4f43af9603b262823a13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fb37ad87a40c7bcbc66109e6ac484" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 43681 } } peers { permanent_uuid: "30991ca8f9ea4f43af9603b262823a13" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 38283 } } peers { permanent_uuid: "40359b022e1b4ec591c7972831f8ce16" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42177 } } }
I20260429 14:08:31.263147 17095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40359b022e1b4ec591c7972831f8ce16 [sys.catalog]: This master's current role is: FOLLOWER
I20260429 14:08:31.263551 17149 mvcc.cc:204] Tried to move back new op lower bound from 7280524129303773184 to 7280524129276645376. Current Snapshot: MvccSnapshot[applied={T|T < 7280524129303773184}]
I20260429 14:08:31.291558 17147 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:08:31.292419 17147 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:08:31.302382 17147 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 30991ca8f9ea4f43af9603b262823a13: Generated new TSK 0
I20260429 14:08:31.302708 17147 catalog_manager.cc:1524] Initializing in-progress tserver states...
Apr 29 14:08:31 dist-test-slave-8n77 krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.2.197.253@KRBTEST.COM
Apr 29 14:08:31 dist-test-slave-8n77 krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.2.197.254@KRBTEST.COM
Apr 29 14:08:31 dist-test-slave-8n77 krb5kdc[16849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.2.197.252@KRBTEST.COM
I20260429 14:08:32.207037 17106 catalog_manager.cc:1269] Loaded cluster ID: 7b1792c4461641a09bab4c6d34e54867
I20260429 14:08:32.207137 17106 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 40359b022e1b4ec591c7972831f8ce16: loading cluster ID for follower catalog manager: success
I20260429 14:08:32.208076 17106 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 40359b022e1b4ec591c7972831f8ce16: acquiring CA information for follower catalog manager: success
I20260429 14:08:32.208416 17106 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 40359b022e1b4ec591c7972831f8ce16: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260429 14:08:32.247452 17143 catalog_manager.cc:1269] Loaded cluster ID: 7b1792c4461641a09bab4c6d34e54867
I20260429 14:08:32.247545 17143 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 910fb37ad87a40c7bcbc66109e6ac484: loading cluster ID for follower catalog manager: success
I20260429 14:08:32.248358 17143 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 910fb37ad87a40c7bcbc66109e6ac484: acquiring CA information for follower catalog manager: success
I20260429 14:08:32.248600 17143 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 910fb37ad87a40c7bcbc66109e6ac484: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260429 14:08:32.430787 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 16865
I20260429 14:08:32.436239 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 16943
I20260429 14:08:32.441293 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 17020
2026-04-29T14:08:32Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4018 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:08:32 dist-test-slave-8n77 krb5kdc[17164](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:08:32 dist-test-slave-8n77 krb5kdc[17164](info): set up 2 sockets
Apr 29 14:08:32 dist-test-slave-8n77 krb5kdc[17164](info): commencing operation
krb5kdc: starting...
W20260429 14:08:34.492748 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:08:34 dist-test-slave-8n77 krb5kdc[17164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:08:34Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:08:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:08:34.646628 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:46767
--webserver_interface=127.2.197.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:36991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:46767
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:08:34.723526 17181 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:08:34.723794 17181 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:08:34.723834 17181 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:08:34.725450 17181 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:08:34.725545 17181 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:08:34.725571 17181 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:08:34.725598 17181 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:08:34.725630 17181 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:08:34.727416 17181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:36991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:46767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:46767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:08:34.727731 17181 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:08:34.727937 17181 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:08:34.730832 17187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:34.730832 17186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:08:34.731035 17181 server_base.cc:1061] running on GCE node
W20260429 14:08:34.730862 17189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:08:34.731527 17181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:08:34.731809 17181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:08:34.732997 17181 hybrid_clock.cc:648] HybridClock initialized: now 1777471714732996 us; error 44 us; skew 500 ppm
Apr 29 14:08:34 dist-test-slave-8n77 krb5kdc[17164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471714, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:08:34.735970 17181 init.cc:377] Logged in from keytab as kudu/127.2.197.254@KRBTEST.COM (short username kudu)
I20260429 14:08:34.736596 17181 webserver.cc:492] Webserver started at http://127.2.197.254:39015/ using document root <none> and password file <none>
I20260429 14:08:34.736804 17181 fs_manager.cc:362] Metadata directory not provided
I20260429 14:08:34.736866 17181 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:08:34.736971 17181 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:08:34.737910 17181 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "65394a78471f40da8e1ab4308542557a"
format_stamp: "Formatted at 2026-04-29 14:08:34 on dist-test-slave-8n77"
server_key: "9730705ee5e7f060e882de1206ca15a0"
server_key_iv: "2307b4cc2042e3d890f84e942e227527"
server_key_version: "encryptionkey@0"
I20260429 14:08:34.738243 17181 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "65394a78471f40da8e1ab4308542557a"
format_stamp: "Formatted at 2026-04-29 14:08:34 on dist-test-slave-8n77"
server_key: "9730705ee5e7f060e882de1206ca15a0"
server_key_iv: "2307b4cc2042e3d890f84e942e227527"
server_key_version: "encryptionkey@0"
I20260429 14:08:34.739535 17181 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:08:34.740310 17196 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:34.740574 17181 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:08:34.740689 17181 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "65394a78471f40da8e1ab4308542557a"
format_stamp: "Formatted at 2026-04-29 14:08:34 on dist-test-slave-8n77"
server_key: "9730705ee5e7f060e882de1206ca15a0"
server_key_iv: "2307b4cc2042e3d890f84e942e227527"
server_key_version: "encryptionkey@0"
I20260429 14:08:34.740772 17181 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:08:34.755959 17181 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:08:34.764243 17181 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:08:34.764415 17181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:08:34.768939 17181 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:46767
I20260429 14:08:34.769017 17249 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:46767 every 8 connection(s)
I20260429 14:08:34.769337 17181 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:08:34.770103 17250 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:08:34.771730 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 17181
I20260429 14:08:34.771822 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:08:34.772014 2839 external_mini_cluster.cc:1468] Setting key bd1a5a74cfcdda4ac2a8f4382ce03f8a
I20260429 14:08:34.773069 17250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65394a78471f40da8e1ab4308542557a: Bootstrap starting.
I20260429 14:08:34.773762 17250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65394a78471f40da8e1ab4308542557a: Neither blocks nor log segments found. Creating new log.
I20260429 14:08:34.774030 17250 log.cc:826] T 00000000000000000000000000000000 P 65394a78471f40da8e1ab4308542557a: Log is configured to *not* fsync() on all Append() calls
I20260429 14:08:34.774798 17250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65394a78471f40da8e1ab4308542557a: No bootstrap required, opened a new log
I20260429 14:08:34.776145 17250 raft_consensus.cc:359] T 00000000000000000000000000000000 P 65394a78471f40da8e1ab4308542557a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65394a78471f40da8e1ab4308542557a" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 46767 } }
I20260429 14:08:34.776304 17250 raft_consensus.cc:385] T 00000000000000000000000000000000 P 65394a78471f40da8e1ab4308542557a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:08:34.776337 17250 raft_consensus.cc:740] T 00000000000000000000000000000000 P 65394a78471f40da8e1ab4308542557a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65394a78471f40da8e1ab4308542557a, State: Initialized, Role: FOLLOWER
I20260429 14:08:34.776496 17250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65394a78471f40da8e1ab4308542557a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65394a78471f40da8e1ab4308542557a" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 46767 } }
I20260429 14:08:34.776605 17250 raft_consensus.cc:399] T 00000000000000000000000000000000 P 65394a78471f40da8e1ab4308542557a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:08:34.776638 17250 raft_consensus.cc:493] T 00000000000000000000000000000000 P 65394a78471f40da8e1ab4308542557a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:08:34.776676 17250 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 65394a78471f40da8e1ab4308542557a [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:08:34.777539 17250 raft_consensus.cc:515] T 00000000000000000000000000000000 P 65394a78471f40da8e1ab4308542557a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65394a78471f40da8e1ab4308542557a" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 46767 } }
I20260429 14:08:34.777676 17250 leader_election.cc:304] T 00000000000000000000000000000000 P 65394a78471f40da8e1ab4308542557a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65394a78471f40da8e1ab4308542557a; no voters:
I20260429 14:08:34.777866 17250 leader_election.cc:290] T 00000000000000000000000000000000 P 65394a78471f40da8e1ab4308542557a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:08:34.777971 17255 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 65394a78471f40da8e1ab4308542557a [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:08:34.778162 17250 sys_catalog.cc:565] T 00000000000000000000000000000000 P 65394a78471f40da8e1ab4308542557a [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:08:34.778173 17255 raft_consensus.cc:697] T 00000000000000000000000000000000 P 65394a78471f40da8e1ab4308542557a [term 1 LEADER]: Becoming Leader. State: Replica: 65394a78471f40da8e1ab4308542557a, State: Running, Role: LEADER
Apr 29 14:08:34 dist-test-slave-8n77 krb5kdc[17164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:08:34.778366 17255 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65394a78471f40da8e1ab4308542557a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65394a78471f40da8e1ab4308542557a" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 46767 } }
I20260429 14:08:34.778716 17256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65394a78471f40da8e1ab4308542557a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65394a78471f40da8e1ab4308542557a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65394a78471f40da8e1ab4308542557a" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 46767 } } }
I20260429 14:08:34.778941 17256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65394a78471f40da8e1ab4308542557a [sys.catalog]: This master's current role is: LEADER
I20260429 14:08:34.778858 17257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65394a78471f40da8e1ab4308542557a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65394a78471f40da8e1ab4308542557a. Latest consensus state: current_term: 1 leader_uuid: "65394a78471f40da8e1ab4308542557a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65394a78471f40da8e1ab4308542557a" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 46767 } } }
I20260429 14:08:34.779152 17257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65394a78471f40da8e1ab4308542557a [sys.catalog]: This master's current role is: LEADER
I20260429 14:08:34.779307 17259 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:08:34.779812 17259 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:08:34.781601 17259 catalog_manager.cc:1357] Generated new cluster ID: c8857cab3d1e467eb7bd3deb6433dc41
I20260429 14:08:34.781652 17259 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:08:34.782725 17253 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:34.772578 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33904 (local address 127.2.197.254:46767)
0429 14:08:34.772793 (+ 215us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:34.772816 (+ 23us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:34.772842 (+ 26us) server_negotiation.cc:408] Connection header received
0429 14:08:34.773227 (+ 385us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:34.773234 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:34.773285 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:34.773434 (+ 149us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:34.774250 (+ 816us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:34.775400 (+ 1150us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:34.776059 (+ 659us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:34.776321 (+ 262us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:34.778950 (+ 2629us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:34.778960 (+ 10us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:34.778960 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:34.778976 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:34.781257 (+ 2281us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:34.781659 (+ 402us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:34.781662 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:34.781663 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:34.781720 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:34.781916 (+ 196us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:34.781917 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:34.781918 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:34.782111 (+ 193us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:34.782279 (+ 168us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:34.782415 (+ 136us) server_negotiation.cc:300] Negotiation successful
0429 14:08:34.782539 (+ 124us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":114,"threads_started":1}
I20260429 14:08:34.783557 2839 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260429 14:08:34.790200 17259 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:08:34.790994 17259 catalog_manager.cc:1514] Loading token signing keys...
Apr 29 14:08:34 dist-test-slave-8n77 krb5kdc[17164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260429 14:08:34.797454 17259 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 65394a78471f40da8e1ab4308542557a: Generated new TSK 0
I20260429 14:08:34.797618 17259 catalog_manager.cc:1524] Initializing in-progress tserver states...
Apr 29 14:08:34 dist-test-slave-8n77 krb5kdc[17164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.2.197.254@KRBTEST.COM
I20260429 14:08:34.805106 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 17181
2026-04-29T14:08:34Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2362 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:08:34 dist-test-slave-8n77 krb5kdc[17280](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:08:34 dist-test-slave-8n77 krb5kdc[17280](info): set up 2 sockets
Apr 29 14:08:34 dist-test-slave-8n77 krb5kdc[17280](info): commencing operation
krb5kdc: starting...
W20260429 14:08:36.827737 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.004s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:08:36 dist-test-slave-8n77 krb5kdc[17280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:08:36Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:08:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:08:36.978593 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:41969
--webserver_interface=127.2.197.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:43751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:41969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:08:37.056689 17297 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:08:37.056967 17297 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:08:37.057013 17297 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:08:37.058668 17297 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:08:37.058770 17297 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:08:37.058794 17297 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:08:37.058815 17297 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:08:37.058845 17297 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:08:37.060674 17297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:43751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:41969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:41969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:08:37.060976 17297 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:08:37.061189 17297 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:08:37.063838 17303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:37.063838 17302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:37.064028 17305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:08:37.064285 17297 server_base.cc:1061] running on GCE node
I20260429 14:08:37.064563 17297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:08:37.064850 17297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:08:37.066043 17297 hybrid_clock.cc:648] HybridClock initialized: now 1777471717066027 us; error 42 us; skew 500 ppm
Apr 29 14:08:37 dist-test-slave-8n77 krb5kdc[17280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471717, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:08:37.068981 17297 init.cc:377] Logged in from keytab as kudu/127.2.197.254@KRBTEST.COM (short username kudu)
I20260429 14:08:37.069610 17297 webserver.cc:492] Webserver started at http://127.2.197.254:35875/ using document root <none> and password file <none>
I20260429 14:08:37.069957 17297 fs_manager.cc:362] Metadata directory not provided
I20260429 14:08:37.070029 17297 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:08:37.070147 17297 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:08:37.071348 17297 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "af9649ac49224c7e91f0508bd22d69b4"
format_stamp: "Formatted at 2026-04-29 14:08:37 on dist-test-slave-8n77"
server_key: "90e5966293ffe503124c0ca266884ffa"
server_key_iv: "436fbd4071e1926980386e57fdfbf19e"
server_key_version: "encryptionkey@0"
I20260429 14:08:37.071725 17297 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "af9649ac49224c7e91f0508bd22d69b4"
format_stamp: "Formatted at 2026-04-29 14:08:37 on dist-test-slave-8n77"
server_key: "90e5966293ffe503124c0ca266884ffa"
server_key_iv: "436fbd4071e1926980386e57fdfbf19e"
server_key_version: "encryptionkey@0"
I20260429 14:08:37.073052 17297 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260429 14:08:37.074090 17312 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:37.074348 17297 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:08:37.074486 17297 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "af9649ac49224c7e91f0508bd22d69b4"
format_stamp: "Formatted at 2026-04-29 14:08:37 on dist-test-slave-8n77"
server_key: "90e5966293ffe503124c0ca266884ffa"
server_key_iv: "436fbd4071e1926980386e57fdfbf19e"
server_key_version: "encryptionkey@0"
I20260429 14:08:37.074582 17297 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:08:37.093394 17297 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:08:37.095539 17297 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:08:37.095736 17297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:08:37.099888 17297 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:41969
I20260429 14:08:37.099941 17365 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:41969 every 8 connection(s)
I20260429 14:08:37.100297 17297 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:08:37.100881 17366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:08:37.103466 17366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4: Bootstrap starting.
I20260429 14:08:37.104061 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 17297
I20260429 14:08:37.104177 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:08:37.104243 17366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4: Neither blocks nor log segments found. Creating new log.
I20260429 14:08:37.104368 2839 external_mini_cluster.cc:1468] Setting key bacfbc48b9d5cf29386626884ca265d0
I20260429 14:08:37.104555 17366 log.cc:826] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4: Log is configured to *not* fsync() on all Append() calls
I20260429 14:08:37.105530 17366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4: No bootstrap required, opened a new log
I20260429 14:08:37.106854 17366 raft_consensus.cc:359] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9649ac49224c7e91f0508bd22d69b4" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 41969 } }
I20260429 14:08:37.106976 17366 raft_consensus.cc:385] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:08:37.106998 17366 raft_consensus.cc:740] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af9649ac49224c7e91f0508bd22d69b4, State: Initialized, Role: FOLLOWER
I20260429 14:08:37.107103 17366 consensus_queue.cc:260] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9649ac49224c7e91f0508bd22d69b4" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 41969 } }
I20260429 14:08:37.107168 17366 raft_consensus.cc:399] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:08:37.107240 17366 raft_consensus.cc:493] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:08:37.107311 17366 raft_consensus.cc:3060] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:08:37.107959 17366 raft_consensus.cc:515] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9649ac49224c7e91f0508bd22d69b4" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 41969 } }
I20260429 14:08:37.108099 17366 leader_election.cc:304] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af9649ac49224c7e91f0508bd22d69b4; no voters:
I20260429 14:08:37.108258 17366 leader_election.cc:290] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:08:37.108321 17371 raft_consensus.cc:2804] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:08:37.108470 17371 raft_consensus.cc:697] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [term 1 LEADER]: Becoming Leader. State: Replica: af9649ac49224c7e91f0508bd22d69b4, State: Running, Role: LEADER
I20260429 14:08:37.108505 17366 sys_catalog.cc:565] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:08:37.108632 17371 consensus_queue.cc:237] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9649ac49224c7e91f0508bd22d69b4" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 41969 } }
I20260429 14:08:37.108978 17372 sys_catalog.cc:455] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af9649ac49224c7e91f0508bd22d69b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9649ac49224c7e91f0508bd22d69b4" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 41969 } } }
I20260429 14:08:37.109141 17372 sys_catalog.cc:458] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [sys.catalog]: This master's current role is: LEADER
I20260429 14:08:37.108975 17371 sys_catalog.cc:455] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af9649ac49224c7e91f0508bd22d69b4. Latest consensus state: current_term: 1 leader_uuid: "af9649ac49224c7e91f0508bd22d69b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9649ac49224c7e91f0508bd22d69b4" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 41969 } } }
I20260429 14:08:37.109305 17371 sys_catalog.cc:458] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [sys.catalog]: This master's current role is: LEADER
I20260429 14:08:37.109520 17377 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:08:37.110080 17377 catalog_manager.cc:1494] Initializing Kudu cluster ID...
Apr 29 14:08:37 dist-test-slave-8n77 krb5kdc[17280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:08:37.111589 17377 catalog_manager.cc:1357] Generated new cluster ID: 92e01676e16043d284bee230b2910f0c
I20260429 14:08:37.111647 17377 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:08:37.115264 17370 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:37.105165 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:43704 (local address 127.2.197.254:41969)
0429 14:08:37.105383 (+ 218us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:37.105394 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:37.105427 (+ 33us) server_negotiation.cc:408] Connection header received
0429 14:08:37.105810 (+ 383us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:37.105816 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:37.105856 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:37.106018 (+ 162us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:37.106913 (+ 895us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:37.107870 (+ 957us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:37.108680 (+ 810us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:37.108936 (+ 256us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:37.111992 (+ 3056us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:37.111997 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:37.111998 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:37.112009 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:37.113671 (+ 1662us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:37.114051 (+ 380us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:37.114054 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:37.114054 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:37.114103 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:37.114360 (+ 257us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:37.114361 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:37.114361 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:37.114587 (+ 226us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:37.114773 (+ 186us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:37.114915 (+ 142us) server_negotiation.cc:300] Negotiation successful
0429 14:08:37.115051 (+ 136us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":101,"threads_started":1}
I20260429 14:08:37.117499 17377 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:08:37.118178 17377 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:08:37.124265 17377 catalog_manager.cc:6044] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4: Generated new TSK 0
I20260429 14:08:37.124514 17377 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:08:37.186777 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.193:0
--local_ip_for_outbound_sockets=127.2.197.193
--webserver_interface=127.2.197.193
--webserver_port=0
--tserver_master_addrs=127.2.197.254:41969
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:43751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:08:37.269570 17394 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:08:37.269822 17394 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:08:37.269865 17394 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:08:37.271734 17394 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:08:37.271837 17394 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:08:37.271890 17394 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.193
I20260429 14:08:37.273743 17394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:43751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.197.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:41969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.193
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:08:37.274036 17394 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:08:37.274241 17394 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:08:37.276962 17402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:37.277024 17399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:37.276962 17400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:08:37.277716 17394 server_base.cc:1061] running on GCE node
I20260429 14:08:37.277907 17394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:08:37.278157 17394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:08:37.279335 17394 hybrid_clock.cc:648] HybridClock initialized: now 1777471717279254 us; error 91 us; skew 500 ppm
Apr 29 14:08:37 dist-test-slave-8n77 krb5kdc[17280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471717, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:08:37.282083 17394 init.cc:377] Logged in from keytab as kudu/127.2.197.193@KRBTEST.COM (short username kudu)
I20260429 14:08:37.282779 17394 webserver.cc:492] Webserver started at http://127.2.197.193:43689/ using document root <none> and password file <none>
I20260429 14:08:37.282986 17394 fs_manager.cc:362] Metadata directory not provided
I20260429 14:08:37.283027 17394 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:08:37.283139 17394 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:08:37.284113 17394 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/data/instance:
uuid: "cde63f442d164697b92f79a9c82b955a"
format_stamp: "Formatted at 2026-04-29 14:08:37 on dist-test-slave-8n77"
server_key: "38689aa1c0fb4b189491af6bab1b4580"
server_key_iv: "ff49f67e55bef8208152272e9b77cdcd"
server_key_version: "encryptionkey@0"
I20260429 14:08:37.284448 17394 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance:
uuid: "cde63f442d164697b92f79a9c82b955a"
format_stamp: "Formatted at 2026-04-29 14:08:37 on dist-test-slave-8n77"
server_key: "38689aa1c0fb4b189491af6bab1b4580"
server_key_iv: "ff49f67e55bef8208152272e9b77cdcd"
server_key_version: "encryptionkey@0"
I20260429 14:08:37.285976 17394 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260429 14:08:37.287227 17409 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:37.287662 17394 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:08:37.287772 17394 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/wal
uuid: "cde63f442d164697b92f79a9c82b955a"
format_stamp: "Formatted at 2026-04-29 14:08:37 on dist-test-slave-8n77"
server_key: "38689aa1c0fb4b189491af6bab1b4580"
server_key_iv: "ff49f67e55bef8208152272e9b77cdcd"
server_key_version: "encryptionkey@0"
I20260429 14:08:37.287868 17394 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:08:37.310572 17394 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:08:37.312927 17394 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:08:37.313094 17394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:08:37.313335 17394 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:08:37.313656 17394 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:08:37.313709 17394 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:37.313755 17394 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:08:37.313779 17394 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:37.319880 17394 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.193:34871
I20260429 14:08:37.319970 17523 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.193:34871 every 8 connection(s)
I20260429 14:08:37.320292 17394 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
I20260429 14:08:37.322067 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 17394
I20260429 14:08:37.322157 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance
I20260429 14:08:37.322372 2839 external_mini_cluster.cc:1468] Setting key 1242b08bead16132bebb854181316faa
Apr 29 14:08:37 dist-test-slave-8n77 krb5kdc[17280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471717, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:08:37.330193 17370 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:37.321091 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:58879 (local address 127.2.197.254:41969)
0429 14:08:37.321139 (+ 48us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:37.321141 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:37.321304 (+ 163us) server_negotiation.cc:408] Connection header received
0429 14:08:37.321871 (+ 567us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:37.321872 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:37.321886 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:37.321984 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:37.322918 (+ 934us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:37.323441 (+ 523us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:37.324376 (+ 935us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:37.324607 (+ 231us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:37.327297 (+ 2690us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:37.327305 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:37.327306 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:37.327322 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:37.328803 (+ 1481us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:37.329272 (+ 469us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:37.329273 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:37.329273 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:37.329305 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:37.329515 (+ 210us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:37.329516 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:37.329516 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:37.329629 (+ 113us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:37.329707 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:37.329991 (+ 284us) server_negotiation.cc:300] Negotiation successful
0429 14:08:37.330093 (+ 102us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260429 14:08:37.330359 17526 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:37.321048 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.193:58879)
0429 14:08:37.321199 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:37.321233 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:37.321731 (+ 498us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:37.322040 (+ 309us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:37.322044 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:37.322548 (+ 504us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:08:37.322828 (+ 280us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:37.322832 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:37.323536 (+ 704us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:37.323537 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:37.324218 (+ 681us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:37.324220 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:37.324354 (+ 134us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:37.325351 (+ 997us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:08:37.325364 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:08:37.327170 (+ 1806us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:08:37.328922 (+ 1752us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:37.328930 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:37.328932 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:37.329161 (+ 229us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:37.329374 (+ 213us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:37.329375 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:37.329376 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:37.329448 (+ 72us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:37.329697 (+ 249us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:08:37.329702 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:08:37.329837 (+ 135us) client_negotiation.cc:770] Sending connection context
0429 14:08:37.329973 (+ 136us) client_negotiation.cc:241] Negotiation successful
0429 14:08:37.330121 (+ 148us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":88,"threads_started":1}
I20260429 14:08:37.330816 17524 heartbeater.cc:344] Connected to a master server at 127.2.197.254:41969
I20260429 14:08:37.330916 17524 heartbeater.cc:461] Registering TS with master...
I20260429 14:08:37.331125 17524 heartbeater.cc:507] Master 127.2.197.254:41969 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.197.194@KRBTEST.COM; defaulting to no policy
I20260429 14:08:37.331801 17330 ts_manager.cc:194] Registered new tserver with Master: cde63f442d164697b92f79a9c82b955a (127.2.197.193:34871)
I20260429 14:08:37.332782 17330 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.193@KRBTEST.COM'} at 127.2.197.193:58879
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:08:37.383144 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.194:0
--local_ip_for_outbound_sockets=127.2.197.194
--webserver_interface=127.2.197.194
--webserver_port=0
--tserver_master_addrs=127.2.197.254:41969
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:43751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:08:37.464376 17531 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:08:37.464605 17531 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:08:37.464632 17531 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:08:37.466351 17531 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:08:37.466419 17531 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:08:37.466449 17531 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.194
I20260429 14:08:37.468327 17531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:43751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.197.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:41969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.194
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:08:37.468578 17531 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:08:37.468801 17531 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260429 14:08:37.472059 17531 server_base.cc:1061] running on GCE node
W20260429 14:08:37.471984 17537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:37.471992 17539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:37.472015 17536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:08:37.472421 17531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:08:37.472726 17531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:08:37.473922 17531 hybrid_clock.cc:648] HybridClock initialized: now 1777471717473907 us; error 35 us; skew 500 ppm
Apr 29 14:08:37 dist-test-slave-8n77 krb5kdc[17280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471717, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:08:37.476905 17531 init.cc:377] Logged in from keytab as kudu/127.2.197.194@KRBTEST.COM (short username kudu)
I20260429 14:08:37.477501 17531 webserver.cc:492] Webserver started at http://127.2.197.194:39345/ using document root <none> and password file <none>
I20260429 14:08:37.477984 17531 fs_manager.cc:362] Metadata directory not provided
I20260429 14:08:37.478159 17531 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:08:37.478413 17531 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:08:37.479708 17531 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/data/instance:
uuid: "7ecaaa30057f485f90fc8781084c9ef2"
format_stamp: "Formatted at 2026-04-29 14:08:37 on dist-test-slave-8n77"
server_key: "1873e40a0ca76054160f00f6125fd435"
server_key_iv: "60ad4af3288d8fb69c32ab12bab2d918"
server_key_version: "encryptionkey@0"
I20260429 14:08:37.480093 17531 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance:
uuid: "7ecaaa30057f485f90fc8781084c9ef2"
format_stamp: "Formatted at 2026-04-29 14:08:37 on dist-test-slave-8n77"
server_key: "1873e40a0ca76054160f00f6125fd435"
server_key_iv: "60ad4af3288d8fb69c32ab12bab2d918"
server_key_version: "encryptionkey@0"
I20260429 14:08:37.481536 17531 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260429 14:08:37.482832 17546 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:37.483182 17531 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:08:37.483282 17531 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/wal
uuid: "7ecaaa30057f485f90fc8781084c9ef2"
format_stamp: "Formatted at 2026-04-29 14:08:37 on dist-test-slave-8n77"
server_key: "1873e40a0ca76054160f00f6125fd435"
server_key_iv: "60ad4af3288d8fb69c32ab12bab2d918"
server_key_version: "encryptionkey@0"
I20260429 14:08:37.483357 17531 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:08:37.513979 17531 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:08:37.516188 17531 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:08:37.516306 17531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:08:37.516649 17531 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:08:37.516983 17531 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:08:37.517041 17531 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:37.517081 17531 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:08:37.517123 17531 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:37.523139 17531 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.194:40909
I20260429 14:08:37.523226 17660 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.194:40909 every 8 connection(s)
I20260429 14:08:37.523530 17531 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
I20260429 14:08:37.529227 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 17531
I20260429 14:08:37.529408 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance
I20260429 14:08:37.529605 2839 external_mini_cluster.cc:1468] Setting key 3259ce20268d4a7e3c252adc3875fe1f
Apr 29 14:08:37 dist-test-slave-8n77 krb5kdc[17280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471717, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:08:37.534022 17663 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:37.524374 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.194:53855)
0429 14:08:37.524583 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:37.524613 (+ 30us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:37.525081 (+ 468us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:37.525318 (+ 237us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:37.525321 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:37.525687 (+ 366us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:08:37.525939 (+ 252us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:37.525942 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:37.526701 (+ 759us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:37.526702 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:37.527078 (+ 376us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:37.527079 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:37.527173 (+ 94us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:37.528143 (+ 970us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:08:37.528152 (+ 9us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:08:37.530482 (+ 2330us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:08:37.532575 (+ 2093us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:37.532584 (+ 9us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:37.532588 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:37.532931 (+ 343us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:37.533187 (+ 256us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:37.533188 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:37.533188 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:37.533250 (+ 62us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:37.533530 (+ 280us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:08:37.533533 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:08:37.533655 (+ 122us) client_negotiation.cc:770] Sending connection context
0429 14:08:37.533749 (+ 94us) client_negotiation.cc:241] Negotiation successful
0429 14:08:37.533865 (+ 116us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":86,"threads_started":1}
I20260429 14:08:37.534041 17370 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:37.524363 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:53855 (local address 127.2.197.254:41969)
0429 14:08:37.524440 (+ 77us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:37.524442 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:37.524730 (+ 288us) server_negotiation.cc:408] Connection header received
0429 14:08:37.525184 (+ 454us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:37.525186 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:37.525197 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:37.525253 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:37.526063 (+ 810us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:37.526602 (+ 539us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:37.527189 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:37.527315 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:37.530620 (+ 3305us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:37.530624 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:37.530624 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:37.530633 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:37.532451 (+ 1818us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:37.533061 (+ 610us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:37.533063 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:37.533064 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:37.533094 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:37.533327 (+ 233us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:37.533328 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:37.533329 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:37.533461 (+ 132us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:37.533534 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:37.533793 (+ 259us) server_negotiation.cc:300] Negotiation successful
0429 14:08:37.533904 (+ 111us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260429 14:08:37.534902 17661 heartbeater.cc:344] Connected to a master server at 127.2.197.254:41969
I20260429 14:08:37.535089 17661 heartbeater.cc:461] Registering TS with master...
I20260429 14:08:37.535516 17661 heartbeater.cc:507] Master 127.2.197.254:41969 requested a full tablet report, sending...
I20260429 14:08:37.536144 17330 ts_manager.cc:194] Registered new tserver with Master: 7ecaaa30057f485f90fc8781084c9ef2 (127.2.197.194:40909)
I20260429 14:08:37.536762 17330 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.194@KRBTEST.COM'} at 127.2.197.194:53855
WARNING: no policy specified for kudu/127.2.197.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:08:37.590796 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.195:0
--local_ip_for_outbound_sockets=127.2.197.195
--webserver_interface=127.2.197.195
--webserver_port=0
--tserver_master_addrs=127.2.197.254:41969
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:43751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:08:37.674427 17668 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:08:37.674652 17668 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:08:37.674680 17668 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:08:37.676532 17668 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:08:37.676589 17668 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:08:37.676623 17668 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.195
I20260429 14:08:37.678699 17668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:43751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.197.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:41969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.195
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:08:37.679005 17668 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:08:37.679229 17668 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:08:37.682337 17674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:37.682364 17673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:37.682358 17676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:08:37.682758 17668 server_base.cc:1061] running on GCE node
I20260429 14:08:37.682934 17668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:08:37.683135 17668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:08:37.684333 17668 hybrid_clock.cc:648] HybridClock initialized: now 1777471717684297 us; error 55 us; skew 500 ppm
Apr 29 14:08:37 dist-test-slave-8n77 krb5kdc[17280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471717, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:08:37.687079 17668 init.cc:377] Logged in from keytab as kudu/127.2.197.195@KRBTEST.COM (short username kudu)
I20260429 14:08:37.687739 17668 webserver.cc:492] Webserver started at http://127.2.197.195:38219/ using document root <none> and password file <none>
I20260429 14:08:37.687944 17668 fs_manager.cc:362] Metadata directory not provided
I20260429 14:08:37.688004 17668 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:08:37.688105 17668 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:08:37.689697 17668 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/data/instance:
uuid: "bcffd137fdc04dcfbf055e9c4529b211"
format_stamp: "Formatted at 2026-04-29 14:08:37 on dist-test-slave-8n77"
server_key: "25e53cf0c3badb473a63e257bf278ae6"
server_key_iv: "6c0059b5d7a1c3086c5fb24f1b46f208"
server_key_version: "encryptionkey@0"
I20260429 14:08:37.690150 17668 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance:
uuid: "bcffd137fdc04dcfbf055e9c4529b211"
format_stamp: "Formatted at 2026-04-29 14:08:37 on dist-test-slave-8n77"
server_key: "25e53cf0c3badb473a63e257bf278ae6"
server_key_iv: "6c0059b5d7a1c3086c5fb24f1b46f208"
server_key_version: "encryptionkey@0"
I20260429 14:08:37.691789 17668 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:08:37.692827 17683 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:37.693145 17668 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:08:37.693320 17668 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/wal
uuid: "bcffd137fdc04dcfbf055e9c4529b211"
format_stamp: "Formatted at 2026-04-29 14:08:37 on dist-test-slave-8n77"
server_key: "25e53cf0c3badb473a63e257bf278ae6"
server_key_iv: "6c0059b5d7a1c3086c5fb24f1b46f208"
server_key_version: "encryptionkey@0"
I20260429 14:08:37.693464 17668 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:08:37.702663 17668 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:08:37.705611 17668 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:08:37.705850 17668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:08:37.706138 17668 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:08:37.706602 17668 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:08:37.706667 17668 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:37.706734 17668 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:08:37.706779 17668 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:37.713789 17668 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.195:36003
I20260429 14:08:37.713901 17798 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.195:36003 every 8 connection(s)
I20260429 14:08:37.714234 17668 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
I20260429 14:08:37.716147 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 17668
I20260429 14:08:37.716269 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance
I20260429 14:08:37.716495 2839 external_mini_cluster.cc:1468] Setting key 0fcf16dae990f16d1049c87d950da0cc
Apr 29 14:08:37 dist-test-slave-8n77 krb5kdc[17280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471717, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:08:37.725353 17370 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:37.715700 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:56985 (local address 127.2.197.254:41969)
0429 14:08:37.715819 (+ 119us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:37.715822 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:37.716117 (+ 295us) server_negotiation.cc:408] Connection header received
0429 14:08:37.716899 (+ 782us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:37.716901 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:37.716915 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:37.717085 (+ 170us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:37.718048 (+ 963us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:37.718671 (+ 623us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:37.719254 (+ 583us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:37.719435 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:37.722202 (+ 2767us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:37.722208 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:37.722208 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:37.722225 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:37.723766 (+ 1541us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:37.724257 (+ 491us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:37.724259 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:37.724259 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:37.724292 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:37.724673 (+ 381us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:37.724674 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:37.724675 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:37.724791 (+ 116us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:37.724872 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:37.725152 (+ 280us) server_negotiation.cc:300] Negotiation successful
0429 14:08:37.725256 (+ 104us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260429 14:08:37.725549 17801 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:37.715668 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.195:56985)
0429 14:08:37.715948 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:37.715990 (+ 42us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:37.716793 (+ 803us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:37.717135 (+ 342us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:37.717140 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:37.717558 (+ 418us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:08:37.717941 (+ 383us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:37.717946 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:37.718780 (+ 834us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:37.718781 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:37.719169 (+ 388us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:37.719171 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:37.719311 (+ 140us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:37.720298 (+ 987us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:08:37.720307 (+ 9us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:08:37.722081 (+ 1774us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:08:37.723892 (+ 1811us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:37.723903 (+ 11us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:37.723906 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:37.724162 (+ 256us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:37.724359 (+ 197us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:37.724360 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:37.724361 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:37.724551 (+ 190us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:37.724873 (+ 322us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:08:37.724880 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:08:37.725014 (+ 134us) client_negotiation.cc:770] Sending connection context
0429 14:08:37.725133 (+ 119us) client_negotiation.cc:241] Negotiation successful
0429 14:08:37.725286 (+ 153us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":125,"threads_started":1}
I20260429 14:08:37.726231 17799 heartbeater.cc:344] Connected to a master server at 127.2.197.254:41969
I20260429 14:08:37.726445 17799 heartbeater.cc:461] Registering TS with master...
I20260429 14:08:37.726691 17799 heartbeater.cc:507] Master 127.2.197.254:41969 requested a full tablet report, sending...
I20260429 14:08:37.727195 17330 ts_manager.cc:194] Registered new tserver with Master: bcffd137fdc04dcfbf055e9c4529b211 (127.2.197.195:36003)
I20260429 14:08:37.727634 17330 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.195@KRBTEST.COM'} at 127.2.197.195:56985
I20260429 14:08:37.730176 2839 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260429 14:08:37.741263 17370 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:37.731884 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:43710 (local address 127.2.197.254:41969)
0429 14:08:37.731960 (+ 76us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:37.731962 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:37.732000 (+ 38us) server_negotiation.cc:408] Connection header received
0429 14:08:37.732123 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:37.732124 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:37.732135 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:37.732226 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:37.735083 (+ 2857us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:37.735587 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:37.736233 (+ 646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:37.736530 (+ 297us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:37.737594 (+ 1064us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:37.737599 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:37.737599 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:37.737606 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:37.739610 (+ 2004us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:37.740023 (+ 413us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:37.740025 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:37.740025 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:37.740053 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:37.740564 (+ 511us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:37.740565 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:37.740566 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:37.740706 (+ 140us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:37.740813 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:37.741094 (+ 281us) server_negotiation.cc:300] Negotiation successful
0429 14:08:37.741191 (+ 97us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260429 14:08:37.743892 17330 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43710:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 14:08:37.746076 17330 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 14:08:37.754082 17812 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:37.749920 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:34871 (local address 127.0.0.1:41734)
0429 14:08:37.750428 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:37.750440 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:37.750494 (+ 54us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:37.750937 (+ 443us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:37.750938 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:37.751029 (+ 91us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:08:37.751505 (+ 476us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:37.751509 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:37.752740 (+ 1231us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:37.752742 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:37.753643 (+ 901us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:37.753647 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:37.753791 (+ 144us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:37.753854 (+ 63us) client_negotiation.cc:770] Sending connection context
0429 14:08:37.753913 (+ 59us) client_negotiation.cc:241] Negotiation successful
0429 14:08:37.753984 (+ 71us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":98,"threads_started":1}
I20260429 14:08:37.754696 17815 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:37.750219 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41734 (local address 127.2.197.193:34871)
0429 14:08:37.750432 (+ 213us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:37.750442 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:37.750555 (+ 113us) server_negotiation.cc:408] Connection header received
0429 14:08:37.750599 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:37.750601 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:37.750619 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:37.750838 (+ 219us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:08:37.751668 (+ 830us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:37.752535 (+ 867us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:37.753831 (+ 1296us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:37.754495 (+ 664us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:37.754538 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:37.754567 (+ 29us) server_negotiation.cc:300] Negotiation successful
0429 14:08:37.754604 (+ 37us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":120,"threads_started":1}
I20260429 14:08:37.755249 17810 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:37.749677 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:36003 (local address 127.0.0.1:36072)
0429 14:08:37.750089 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:37.750144 (+ 55us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:37.750266 (+ 122us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:37.750592 (+ 326us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:37.750595 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:37.750598 (+ 3us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:08:37.750920 (+ 322us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:37.750923 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:37.752689 (+ 1766us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:37.752690 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:37.753788 (+ 1098us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:37.753792 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:37.755002 (+ 1210us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:37.755007 (+ 5us) client_negotiation.cc:770] Sending connection context
0429 14:08:37.755092 (+ 85us) client_negotiation.cc:241] Negotiation successful
0429 14:08:37.755158 (+ 66us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":80,"threads_started":1}
I20260429 14:08:37.755476 17811 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:37.749747 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36072 (local address 127.2.197.195:36003)
0429 14:08:37.749943 (+ 196us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:37.749953 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:37.750243 (+ 290us) server_negotiation.cc:408] Connection header received
0429 14:08:37.750394 (+ 151us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:37.750397 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:37.750415 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:37.750485 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:08:37.751096 (+ 611us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:37.752469 (+ 1373us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:37.753974 (+ 1505us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:37.754899 (+ 925us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:37.754948 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:37.755174 (+ 226us) server_negotiation.cc:300] Negotiation successful
0429 14:08:37.755245 (+ 71us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":104,"threads_started":1}
I20260429 14:08:37.755538 17457 tablet_service.cc:1511] Processing CreateTablet for tablet 06842361b98a438980f0d442eb328e2d (DEFAULT_TABLE table=test-table [id=512ea896dd6848ae9cdece8f6f054f59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:08:37.755885 17457 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06842361b98a438980f0d442eb328e2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:08:37.756049 17732 tablet_service.cc:1511] Processing CreateTablet for tablet 06842361b98a438980f0d442eb328e2d (DEFAULT_TABLE table=test-table [id=512ea896dd6848ae9cdece8f6f054f59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:08:37.756377 17732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06842361b98a438980f0d442eb328e2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:08:37.756598 17814 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:37.749788 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:40909 (local address 127.0.0.1:35162)
0429 14:08:37.750195 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:37.750225 (+ 30us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:37.750332 (+ 107us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:37.750612 (+ 280us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:37.750613 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:37.750615 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:08:37.750925 (+ 310us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:37.750927 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:37.755272 (+ 4345us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:37.755274 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:37.756325 (+ 1051us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:37.756329 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:37.756434 (+ 105us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:37.756438 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 14:08:37.756479 (+ 41us) client_negotiation.cc:241] Negotiation successful
0429 14:08:37.756525 (+ 46us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":103,"threads_started":1}
I20260429 14:08:37.757290 17813 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:37.749905 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35162 (local address 127.2.197.194:40909)
0429 14:08:37.750101 (+ 196us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:37.750108 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:37.750325 (+ 217us) server_negotiation.cc:408] Connection header received
0429 14:08:37.750438 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:37.750440 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:37.750476 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:37.750533 (+ 57us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:08:37.752826 (+ 2293us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:37.754131 (+ 1305us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:37.756493 (+ 2362us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:37.757078 (+ 585us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:37.757123 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:37.757156 (+ 33us) server_negotiation.cc:300] Negotiation successful
0429 14:08:37.757213 (+ 57us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":81,"threads_started":1}
I20260429 14:08:37.758209 17595 tablet_service.cc:1511] Processing CreateTablet for tablet 06842361b98a438980f0d442eb328e2d (DEFAULT_TABLE table=test-table [id=512ea896dd6848ae9cdece8f6f054f59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:08:37.758512 17595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06842361b98a438980f0d442eb328e2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:08:37.759022 17816 tablet_bootstrap.cc:492] T 06842361b98a438980f0d442eb328e2d P cde63f442d164697b92f79a9c82b955a: Bootstrap starting.
I20260429 14:08:37.759801 17816 tablet_bootstrap.cc:654] T 06842361b98a438980f0d442eb328e2d P cde63f442d164697b92f79a9c82b955a: Neither blocks nor log segments found. Creating new log.
I20260429 14:08:37.760174 17816 log.cc:826] T 06842361b98a438980f0d442eb328e2d P cde63f442d164697b92f79a9c82b955a: Log is configured to *not* fsync() on all Append() calls
I20260429 14:08:37.760550 17817 tablet_bootstrap.cc:492] T 06842361b98a438980f0d442eb328e2d P bcffd137fdc04dcfbf055e9c4529b211: Bootstrap starting.
I20260429 14:08:37.761013 17819 tablet_bootstrap.cc:492] T 06842361b98a438980f0d442eb328e2d P 7ecaaa30057f485f90fc8781084c9ef2: Bootstrap starting.
I20260429 14:08:37.761165 17816 tablet_bootstrap.cc:492] T 06842361b98a438980f0d442eb328e2d P cde63f442d164697b92f79a9c82b955a: No bootstrap required, opened a new log
I20260429 14:08:37.761250 17816 ts_tablet_manager.cc:1403] T 06842361b98a438980f0d442eb328e2d P cde63f442d164697b92f79a9c82b955a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260429 14:08:37.761432 17817 tablet_bootstrap.cc:654] T 06842361b98a438980f0d442eb328e2d P bcffd137fdc04dcfbf055e9c4529b211: Neither blocks nor log segments found. Creating new log.
I20260429 14:08:37.761646 17819 tablet_bootstrap.cc:654] T 06842361b98a438980f0d442eb328e2d P 7ecaaa30057f485f90fc8781084c9ef2: Neither blocks nor log segments found. Creating new log.
I20260429 14:08:37.761787 17817 log.cc:826] T 06842361b98a438980f0d442eb328e2d P bcffd137fdc04dcfbf055e9c4529b211: Log is configured to *not* fsync() on all Append() calls
I20260429 14:08:37.762370 17819 log.cc:826] T 06842361b98a438980f0d442eb328e2d P 7ecaaa30057f485f90fc8781084c9ef2: Log is configured to *not* fsync() on all Append() calls
I20260429 14:08:37.762673 17817 tablet_bootstrap.cc:492] T 06842361b98a438980f0d442eb328e2d P bcffd137fdc04dcfbf055e9c4529b211: No bootstrap required, opened a new log
I20260429 14:08:37.762766 17817 ts_tablet_manager.cc:1403] T 06842361b98a438980f0d442eb328e2d P bcffd137fdc04dcfbf055e9c4529b211: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260429 14:08:37.763118 17819 tablet_bootstrap.cc:492] T 06842361b98a438980f0d442eb328e2d P 7ecaaa30057f485f90fc8781084c9ef2: No bootstrap required, opened a new log
I20260429 14:08:37.763218 17819 ts_tablet_manager.cc:1403] T 06842361b98a438980f0d442eb328e2d P 7ecaaa30057f485f90fc8781084c9ef2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260429 14:08:37.763756 17816 raft_consensus.cc:359] T 06842361b98a438980f0d442eb328e2d P cde63f442d164697b92f79a9c82b955a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcffd137fdc04dcfbf055e9c4529b211" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36003 } } peers { permanent_uuid: "7ecaaa30057f485f90fc8781084c9ef2" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40909 } } peers { permanent_uuid: "cde63f442d164697b92f79a9c82b955a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34871 } }
I20260429 14:08:37.763928 17816 raft_consensus.cc:385] T 06842361b98a438980f0d442eb328e2d P cde63f442d164697b92f79a9c82b955a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:08:37.763963 17816 raft_consensus.cc:740] T 06842361b98a438980f0d442eb328e2d P cde63f442d164697b92f79a9c82b955a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cde63f442d164697b92f79a9c82b955a, State: Initialized, Role: FOLLOWER
I20260429 14:08:37.764075 17816 consensus_queue.cc:260] T 06842361b98a438980f0d442eb328e2d P cde63f442d164697b92f79a9c82b955a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcffd137fdc04dcfbf055e9c4529b211" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36003 } } peers { permanent_uuid: "7ecaaa30057f485f90fc8781084c9ef2" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40909 } } peers { permanent_uuid: "cde63f442d164697b92f79a9c82b955a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34871 } }
I20260429 14:08:37.764339 17816 ts_tablet_manager.cc:1434] T 06842361b98a438980f0d442eb328e2d P cde63f442d164697b92f79a9c82b955a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260429 14:08:37.764245 17817 raft_consensus.cc:359] T 06842361b98a438980f0d442eb328e2d P bcffd137fdc04dcfbf055e9c4529b211 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcffd137fdc04dcfbf055e9c4529b211" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36003 } } peers { permanent_uuid: "7ecaaa30057f485f90fc8781084c9ef2" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40909 } } peers { permanent_uuid: "cde63f442d164697b92f79a9c82b955a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34871 } }
I20260429 14:08:37.764397 17817 raft_consensus.cc:385] T 06842361b98a438980f0d442eb328e2d P bcffd137fdc04dcfbf055e9c4529b211 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:08:37.764425 17817 raft_consensus.cc:740] T 06842361b98a438980f0d442eb328e2d P bcffd137fdc04dcfbf055e9c4529b211 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcffd137fdc04dcfbf055e9c4529b211, State: Initialized, Role: FOLLOWER
I20260429 14:08:37.764467 17524 heartbeater.cc:499] Master 127.2.197.254:41969 was elected leader, sending a full tablet report...
I20260429 14:08:37.764509 17817 consensus_queue.cc:260] T 06842361b98a438980f0d442eb328e2d P bcffd137fdc04dcfbf055e9c4529b211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcffd137fdc04dcfbf055e9c4529b211" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36003 } } peers { permanent_uuid: "7ecaaa30057f485f90fc8781084c9ef2" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40909 } } peers { permanent_uuid: "cde63f442d164697b92f79a9c82b955a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34871 } }
I20260429 14:08:37.764732 17817 ts_tablet_manager.cc:1434] T 06842361b98a438980f0d442eb328e2d P bcffd137fdc04dcfbf055e9c4529b211: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260429 14:08:37.764856 17799 heartbeater.cc:499] Master 127.2.197.254:41969 was elected leader, sending a full tablet report...
I20260429 14:08:37.766024 17819 raft_consensus.cc:359] T 06842361b98a438980f0d442eb328e2d P 7ecaaa30057f485f90fc8781084c9ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcffd137fdc04dcfbf055e9c4529b211" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36003 } } peers { permanent_uuid: "7ecaaa30057f485f90fc8781084c9ef2" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40909 } } peers { permanent_uuid: "cde63f442d164697b92f79a9c82b955a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34871 } }
I20260429 14:08:37.766144 17819 raft_consensus.cc:385] T 06842361b98a438980f0d442eb328e2d P 7ecaaa30057f485f90fc8781084c9ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:08:37.766167 17819 raft_consensus.cc:740] T 06842361b98a438980f0d442eb328e2d P 7ecaaa30057f485f90fc8781084c9ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ecaaa30057f485f90fc8781084c9ef2, State: Initialized, Role: FOLLOWER
I20260429 14:08:37.766311 17819 consensus_queue.cc:260] T 06842361b98a438980f0d442eb328e2d P 7ecaaa30057f485f90fc8781084c9ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcffd137fdc04dcfbf055e9c4529b211" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36003 } } peers { permanent_uuid: "7ecaaa30057f485f90fc8781084c9ef2" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40909 } } peers { permanent_uuid: "cde63f442d164697b92f79a9c82b955a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34871 } }
I20260429 14:08:37.766743 17819 ts_tablet_manager.cc:1434] T 06842361b98a438980f0d442eb328e2d P 7ecaaa30057f485f90fc8781084c9ef2: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260429 14:08:37.766872 17661 heartbeater.cc:499] Master 127.2.197.254:41969 was elected leader, sending a full tablet report...
W20260429 14:08:37.774219 17662 tablet.cc:2404] T 06842361b98a438980f0d442eb328e2d P 7ecaaa30057f485f90fc8781084c9ef2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260429 14:08:37.821101 17525 tablet.cc:2404] T 06842361b98a438980f0d442eb328e2d P cde63f442d164697b92f79a9c82b955a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260429 14:08:37.965126 17800 tablet.cc:2404] T 06842361b98a438980f0d442eb328e2d P bcffd137fdc04dcfbf055e9c4529b211: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 14:08:37.965359 17823 raft_consensus.cc:493] T 06842361b98a438980f0d442eb328e2d P bcffd137fdc04dcfbf055e9c4529b211 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:08:37.965512 17823 raft_consensus.cc:515] T 06842361b98a438980f0d442eb328e2d P bcffd137fdc04dcfbf055e9c4529b211 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcffd137fdc04dcfbf055e9c4529b211" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36003 } } peers { permanent_uuid: "7ecaaa30057f485f90fc8781084c9ef2" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40909 } } peers { permanent_uuid: "cde63f442d164697b92f79a9c82b955a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34871 } }
I20260429 14:08:37.966192 17823 leader_election.cc:290] T 06842361b98a438980f0d442eb328e2d P bcffd137fdc04dcfbf055e9c4529b211 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ecaaa30057f485f90fc8781084c9ef2 (127.2.197.194:40909), cde63f442d164697b92f79a9c82b955a (127.2.197.193:34871)
I20260429 14:08:37.968773 17801 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:37.966528 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:34871 (local address 127.2.197.195:46299)
0429 14:08:37.966649 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:37.966663 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:37.966711 (+ 48us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:37.966902 (+ 191us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:37.966903 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:37.966906 (+ 3us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:08:37.967130 (+ 224us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:37.967132 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:37.967924 (+ 792us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:37.967925 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:37.968602 (+ 677us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:37.968604 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:37.968665 (+ 61us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:37.968669 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 14:08:37.968693 (+ 24us) client_negotiation.cc:241] Negotiation successful
0429 14:08:37.968720 (+ 27us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260429 14:08:37.969318 17815 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:37.966512 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:46299 (local address 127.2.197.193:34871)
0429 14:08:37.966662 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:37.966665 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:37.966722 (+ 57us) server_negotiation.cc:408] Connection header received
0429 14:08:37.966801 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:37.966803 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:37.966814 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:37.966878 (+ 64us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:08:37.967229 (+ 351us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:37.967813 (+ 584us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:37.968715 (+ 902us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:37.969186 (+ 471us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:37.969195 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:37.969218 (+ 23us) server_negotiation.cc:300] Negotiation successful
0429 14:08:37.969248 (+ 30us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260429 14:08:37.969684 17825 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:37.966541 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:40909 (local address 127.2.197.195:53583)
0429 14:08:37.966743 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:37.966754 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:37.966814 (+ 60us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:37.966992 (+ 178us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:37.966993 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:37.966995 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:08:37.967277 (+ 282us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:37.967280 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:37.968121 (+ 841us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:37.968122 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:37.968851 (+ 729us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:37.968853 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:37.969459 (+ 606us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:37.969463 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 14:08:37.969568 (+ 105us) client_negotiation.cc:241] Negotiation successful
0429 14:08:37.969617 (+ 49us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":129,"threads_started":1}
I20260429 14:08:37.969748 17813 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:37.966753 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:53583 (local address 127.2.197.194:40909)
0429 14:08:37.966800 (+ 47us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:37.966802 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:37.966811 (+ 9us) server_negotiation.cc:408] Connection header received
0429 14:08:37.966904 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:37.966905 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:37.966915 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:37.966981 (+ 66us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:08:37.967449 (+ 468us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:37.968010 (+ 561us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:37.968961 (+ 951us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:37.969404 (+ 443us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:37.969422 (+ 18us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:37.969616 (+ 194us) server_negotiation.cc:300] Negotiation successful
0429 14:08:37.969670 (+ 54us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260429 14:08:37.969709 17478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06842361b98a438980f0d442eb328e2d" candidate_uuid: "bcffd137fdc04dcfbf055e9c4529b211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cde63f442d164697b92f79a9c82b955a" is_pre_election: true
I20260429 14:08:37.969882 17478 raft_consensus.cc:2468] T 06842361b98a438980f0d442eb328e2d P cde63f442d164697b92f79a9c82b955a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcffd137fdc04dcfbf055e9c4529b211 in term 0.
I20260429 14:08:37.970197 17685 leader_election.cc:304] T 06842361b98a438980f0d442eb328e2d P bcffd137fdc04dcfbf055e9c4529b211 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcffd137fdc04dcfbf055e9c4529b211, cde63f442d164697b92f79a9c82b955a; no voters:
I20260429 14:08:37.970196 17615 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06842361b98a438980f0d442eb328e2d" candidate_uuid: "bcffd137fdc04dcfbf055e9c4529b211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ecaaa30057f485f90fc8781084c9ef2" is_pre_election: true
I20260429 14:08:37.970367 17823 raft_consensus.cc:2804] T 06842361b98a438980f0d442eb328e2d P bcffd137fdc04dcfbf055e9c4529b211 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 14:08:37.970357 17615 raft_consensus.cc:2468] T 06842361b98a438980f0d442eb328e2d P 7ecaaa30057f485f90fc8781084c9ef2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcffd137fdc04dcfbf055e9c4529b211 in term 0.
I20260429 14:08:37.970531 17823 raft_consensus.cc:493] T 06842361b98a438980f0d442eb328e2d P bcffd137fdc04dcfbf055e9c4529b211 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 14:08:37.970592 17823 raft_consensus.cc:3060] T 06842361b98a438980f0d442eb328e2d P bcffd137fdc04dcfbf055e9c4529b211 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:08:37.971458 17823 raft_consensus.cc:515] T 06842361b98a438980f0d442eb328e2d P bcffd137fdc04dcfbf055e9c4529b211 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcffd137fdc04dcfbf055e9c4529b211" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36003 } } peers { permanent_uuid: "7ecaaa30057f485f90fc8781084c9ef2" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40909 } } peers { permanent_uuid: "cde63f442d164697b92f79a9c82b955a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34871 } }
I20260429 14:08:37.971675 17823 leader_election.cc:290] T 06842361b98a438980f0d442eb328e2d P bcffd137fdc04dcfbf055e9c4529b211 [CANDIDATE]: Term 1 election: Requested vote from peers 7ecaaa30057f485f90fc8781084c9ef2 (127.2.197.194:40909), cde63f442d164697b92f79a9c82b955a (127.2.197.193:34871)
I20260429 14:08:37.971864 17615 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06842361b98a438980f0d442eb328e2d" candidate_uuid: "bcffd137fdc04dcfbf055e9c4529b211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ecaaa30057f485f90fc8781084c9ef2"
I20260429 14:08:37.971863 17478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06842361b98a438980f0d442eb328e2d" candidate_uuid: "bcffd137fdc04dcfbf055e9c4529b211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cde63f442d164697b92f79a9c82b955a"
I20260429 14:08:37.971961 17615 raft_consensus.cc:3060] T 06842361b98a438980f0d442eb328e2d P 7ecaaa30057f485f90fc8781084c9ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:08:37.971958 17478 raft_consensus.cc:3060] T 06842361b98a438980f0d442eb328e2d P cde63f442d164697b92f79a9c82b955a [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:08:37.972705 17615 raft_consensus.cc:2468] T 06842361b98a438980f0d442eb328e2d P 7ecaaa30057f485f90fc8781084c9ef2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcffd137fdc04dcfbf055e9c4529b211 in term 1.
I20260429 14:08:37.972846 17478 raft_consensus.cc:2468] T 06842361b98a438980f0d442eb328e2d P cde63f442d164697b92f79a9c82b955a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcffd137fdc04dcfbf055e9c4529b211 in term 1.
I20260429 14:08:37.972939 17687 leader_election.cc:304] T 06842361b98a438980f0d442eb328e2d P bcffd137fdc04dcfbf055e9c4529b211 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecaaa30057f485f90fc8781084c9ef2, bcffd137fdc04dcfbf055e9c4529b211; no voters:
I20260429 14:08:37.973037 17823 raft_consensus.cc:2804] T 06842361b98a438980f0d442eb328e2d P bcffd137fdc04dcfbf055e9c4529b211 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:08:37.973212 17823 raft_consensus.cc:697] T 06842361b98a438980f0d442eb328e2d P bcffd137fdc04dcfbf055e9c4529b211 [term 1 LEADER]: Becoming Leader. State: Replica: bcffd137fdc04dcfbf055e9c4529b211, State: Running, Role: LEADER
I20260429 14:08:37.973356 17823 consensus_queue.cc:237] T 06842361b98a438980f0d442eb328e2d P bcffd137fdc04dcfbf055e9c4529b211 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcffd137fdc04dcfbf055e9c4529b211" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36003 } } peers { permanent_uuid: "7ecaaa30057f485f90fc8781084c9ef2" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40909 } } peers { permanent_uuid: "cde63f442d164697b92f79a9c82b955a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34871 } }
I20260429 14:08:37.974447 17330 catalog_manager.cc:5671] T 06842361b98a438980f0d442eb328e2d P bcffd137fdc04dcfbf055e9c4529b211 reported cstate change: term changed from 0 to 1, leader changed from <none> to bcffd137fdc04dcfbf055e9c4529b211 (127.2.197.195). New cstate: current_term: 1 leader_uuid: "bcffd137fdc04dcfbf055e9c4529b211" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcffd137fdc04dcfbf055e9c4529b211" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ecaaa30057f485f90fc8781084c9ef2" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cde63f442d164697b92f79a9c82b955a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34871 } health_report { overall_health: UNKNOWN } } }
I20260429 14:08:37.989122 17811 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:37.986470 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36084 (local address 127.2.197.195:36003)
0429 14:08:37.986599 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:37.986601 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:37.986610 (+ 9us) server_negotiation.cc:408] Connection header received
0429 14:08:37.986686 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:37.986687 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:37.986705 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:37.986766 (+ 61us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:08:37.987352 (+ 586us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:37.987868 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:37.988505 (+ 637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:37.988666 (+ 161us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:37.988705 (+ 39us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:08:37.988804 (+ 99us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:08:37.988875 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:37.988961 (+ 86us) server_negotiation.cc:300] Negotiation successful
0429 14:08:37.988984 (+ 23us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260429 14:08:37.991222 17615 raft_consensus.cc:1275] T 06842361b98a438980f0d442eb328e2d P 7ecaaa30057f485f90fc8781084c9ef2 [term 1 FOLLOWER]: Refusing update from remote peer bcffd137fdc04dcfbf055e9c4529b211: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:08:37.991242 17478 raft_consensus.cc:1275] T 06842361b98a438980f0d442eb328e2d P cde63f442d164697b92f79a9c82b955a [term 1 FOLLOWER]: Refusing update from remote peer bcffd137fdc04dcfbf055e9c4529b211: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:08:37.991716 17823 consensus_queue.cc:1048] T 06842361b98a438980f0d442eb328e2d P bcffd137fdc04dcfbf055e9c4529b211 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecaaa30057f485f90fc8781084c9ef2" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:08:37.991873 17826 consensus_queue.cc:1048] T 06842361b98a438980f0d442eb328e2d P bcffd137fdc04dcfbf055e9c4529b211 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cde63f442d164697b92f79a9c82b955a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:08:37.994566 17830 mvcc.cc:204] Tried to move back new op lower bound from 7280524156889366528 to 7280524156820025344. Current Snapshot: MvccSnapshot[applied={T|T < 7280524156889366528}]
I20260429 14:08:37.995385 17831 mvcc.cc:204] Tried to move back new op lower bound from 7280524156889366528 to 7280524156820025344. Current Snapshot: MvccSnapshot[applied={T|T < 7280524156889366528}]
I20260429 14:08:37.996776 17330 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:08:37.996886 17330 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:08:37.998086 17330 catalog_manager.cc:5958] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4: Sending DeleteTablet for 3 replicas of tablet 06842361b98a438980f0d442eb328e2d
I20260429 14:08:37.998586 17457 tablet_service.cc:1558] Processing DeleteTablet for tablet 06842361b98a438980f0d442eb328e2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:08:37 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:41734
I20260429 14:08:37.998610 17595 tablet_service.cc:1558] Processing DeleteTablet for tablet 06842361b98a438980f0d442eb328e2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:08:37 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:35162
I20260429 14:08:37.998883 17839 tablet_replica.cc:333] T 06842361b98a438980f0d442eb328e2d P 7ecaaa30057f485f90fc8781084c9ef2: stopping tablet replica
I20260429 14:08:37.998898 17838 tablet_replica.cc:333] T 06842361b98a438980f0d442eb328e2d P cde63f442d164697b92f79a9c82b955a: stopping tablet replica
I20260429 14:08:37.998888 17732 tablet_service.cc:1558] Processing DeleteTablet for tablet 06842361b98a438980f0d442eb328e2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:08:37 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:36072
I20260429 14:08:37.998988 17838 raft_consensus.cc:2243] T 06842361b98a438980f0d442eb328e2d P cde63f442d164697b92f79a9c82b955a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:08:37.999011 17839 raft_consensus.cc:2243] T 06842361b98a438980f0d442eb328e2d P 7ecaaa30057f485f90fc8781084c9ef2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:08:37.999092 17839 raft_consensus.cc:2272] T 06842361b98a438980f0d442eb328e2d P 7ecaaa30057f485f90fc8781084c9ef2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:08:37.999089 17838 raft_consensus.cc:2272] T 06842361b98a438980f0d442eb328e2d P cde63f442d164697b92f79a9c82b955a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:08:37.999313 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 17394
I20260429 14:08:37.999644 17839 ts_tablet_manager.cc:1916] T 06842361b98a438980f0d442eb328e2d P 7ecaaa30057f485f90fc8781084c9ef2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:08:38.001830 17839 ts_tablet_manager.cc:1929] T 06842361b98a438980f0d442eb328e2d P 7ecaaa30057f485f90fc8781084c9ef2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 14:08:38.001963 17839 log.cc:1199] T 06842361b98a438980f0d442eb328e2d P 7ecaaa30057f485f90fc8781084c9ef2: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/wal/wals/06842361b98a438980f0d442eb328e2d
I20260429 14:08:38.002264 17839 ts_tablet_manager.cc:1950] T 06842361b98a438980f0d442eb328e2d P 7ecaaa30057f485f90fc8781084c9ef2: Deleting consensus metadata
I20260429 14:08:38.002854 17316 catalog_manager.cc:5002] TS 7ecaaa30057f485f90fc8781084c9ef2 (127.2.197.194:40909): tablet 06842361b98a438980f0d442eb328e2d (table test-table [id=512ea896dd6848ae9cdece8f6f054f59]) successfully deleted
I20260429 14:08:38.006507 17840 tablet_replica.cc:333] T 06842361b98a438980f0d442eb328e2d P bcffd137fdc04dcfbf055e9c4529b211: stopping tablet replica
W20260429 14:08:38.006642 17314 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.197.193:34871 (error 108)
I20260429 14:08:38.006692 17840 raft_consensus.cc:2243] T 06842361b98a438980f0d442eb328e2d P bcffd137fdc04dcfbf055e9c4529b211 [term 1 LEADER]: Raft consensus shutting down.
I20260429 14:08:38.006815 17840 raft_consensus.cc:2272] T 06842361b98a438980f0d442eb328e2d P bcffd137fdc04dcfbf055e9c4529b211 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:08:38.007165 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 17531
I20260429 14:08:38.007324 17814 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:38.007149 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:34871 (local address 127.0.0.1:41746)
0429 14:08:38.007242 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:38.007283 (+ 41us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.193:34871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260429 14:08:38.008424 17840 ts_tablet_manager.cc:1916] T 06842361b98a438980f0d442eb328e2d P bcffd137fdc04dcfbf055e9c4529b211: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:08:38.010313 17840 ts_tablet_manager.cc:1929] T 06842361b98a438980f0d442eb328e2d P bcffd137fdc04dcfbf055e9c4529b211: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 14:08:38.010401 17840 log.cc:1199] T 06842361b98a438980f0d442eb328e2d P bcffd137fdc04dcfbf055e9c4529b211: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/wal/wals/06842361b98a438980f0d442eb328e2d
I20260429 14:08:38.010694 17840 ts_tablet_manager.cc:1950] T 06842361b98a438980f0d442eb328e2d P bcffd137fdc04dcfbf055e9c4529b211: Deleting consensus metadata
I20260429 14:08:38.014474 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 17668
W20260429 14:08:38.014593 17314 catalog_manager.cc:4729] TS cde63f442d164697b92f79a9c82b955a (127.2.197.193:34871): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 06842361b98a438980f0d442eb328e2d: Network error: Client connection negotiation failed: client connection to 127.2.197.193:34871: connect: Connection refused (error 111)
I20260429 14:08:38.014905 17314 catalog_manager.cc:5002] TS bcffd137fdc04dcfbf055e9c4529b211 (127.2.197.195:36003): tablet 06842361b98a438980f0d442eb328e2d (table test-table [id=512ea896dd6848ae9cdece8f6f054f59]) successfully deleted
I20260429 14:08:38.022626 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 17297
I20260429 14:08:38.028475 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:41969
--webserver_interface=127.2.197.254
--webserver_port=35875
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:43751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:41969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:08:38.111536 17843 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:08:38.111797 17843 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:08:38.111837 17843 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:08:38.113642 17843 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:08:38.113754 17843 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:08:38.113783 17843 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:08:38.113808 17843 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:08:38.113839 17843 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:08:38.115871 17843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:43751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:41969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:41969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=35875
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:08:38.116204 17843 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:08:38.116432 17843 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:08:38.122992 17852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:38.122992 17849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:38.122992 17850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:08:38.123195 17843 server_base.cc:1061] running on GCE node
I20260429 14:08:38.123607 17843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:08:38.123901 17843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:08:38.125064 17843 hybrid_clock.cc:648] HybridClock initialized: now 1777471718125051 us; error 33 us; skew 500 ppm
Apr 29 14:08:38 dist-test-slave-8n77 krb5kdc[17280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471718, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:08:38.127704 17843 init.cc:377] Logged in from keytab as kudu/127.2.197.254@KRBTEST.COM (short username kudu)
I20260429 14:08:38.128279 17843 webserver.cc:492] Webserver started at http://127.2.197.254:35875/ using document root <none> and password file <none>
I20260429 14:08:38.128531 17843 fs_manager.cc:362] Metadata directory not provided
I20260429 14:08:38.128607 17843 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:08:38.129866 17843 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:08:38.130587 17859 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:38.130789 17843 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:08:38.130867 17843 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "af9649ac49224c7e91f0508bd22d69b4"
format_stamp: "Formatted at 2026-04-29 14:08:37 on dist-test-slave-8n77"
server_key: "90e5966293ffe503124c0ca266884ffa"
server_key_iv: "436fbd4071e1926980386e57fdfbf19e"
server_key_version: "encryptionkey@0"
I20260429 14:08:38.131222 17843 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:08:38.139777 17843 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:08:38.141781 17843 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:08:38.141955 17843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:08:38.146420 17843 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:41969
I20260429 14:08:38.146481 17911 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:41969 every 8 connection(s)
I20260429 14:08:38.146843 17843 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:08:38.147941 17912 sys_catalog.cc:263] Verifying existing consensus state
I20260429 14:08:38.148595 17912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4: Bootstrap starting.
I20260429 14:08:38.154487 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 17843
I20260429 14:08:38.154866 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.193:34871
--local_ip_for_outbound_sockets=127.2.197.193
--tserver_master_addrs=127.2.197.254:41969
--webserver_port=43689
--webserver_interface=127.2.197.193
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:43751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260429 14:08:38.158844 17912 log.cc:826] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4: Log is configured to *not* fsync() on all Append() calls
I20260429 14:08:38.160975 17912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260429 14:08:38.161336 17912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4: Bootstrap complete.
I20260429 14:08:38.164286 17912 raft_consensus.cc:359] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9649ac49224c7e91f0508bd22d69b4" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 41969 } }
I20260429 14:08:38.164624 17912 raft_consensus.cc:740] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: af9649ac49224c7e91f0508bd22d69b4, State: Initialized, Role: FOLLOWER
I20260429 14:08:38.164798 17912 consensus_queue.cc:260] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9649ac49224c7e91f0508bd22d69b4" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 41969 } }
I20260429 14:08:38.164896 17912 raft_consensus.cc:399] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:08:38.164970 17912 raft_consensus.cc:493] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:08:38.165023 17912 raft_consensus.cc:3060] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [term 1 FOLLOWER]: Advancing to term 2
I20260429 14:08:38.165980 17912 raft_consensus.cc:515] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9649ac49224c7e91f0508bd22d69b4" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 41969 } }
I20260429 14:08:38.166123 17912 leader_election.cc:304] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af9649ac49224c7e91f0508bd22d69b4; no voters:
I20260429 14:08:38.166347 17912 leader_election.cc:290] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260429 14:08:38.166388 17916 raft_consensus.cc:2804] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [term 2 FOLLOWER]: Leader election won for term 2
I20260429 14:08:38.166504 17916 raft_consensus.cc:697] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [term 2 LEADER]: Becoming Leader. State: Replica: af9649ac49224c7e91f0508bd22d69b4, State: Running, Role: LEADER
I20260429 14:08:38.166585 17916 consensus_queue.cc:237] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9649ac49224c7e91f0508bd22d69b4" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 41969 } }
I20260429 14:08:38.166615 17912 sys_catalog.cc:565] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:08:38.166929 17918 sys_catalog.cc:455] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "af9649ac49224c7e91f0508bd22d69b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9649ac49224c7e91f0508bd22d69b4" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 41969 } } }
I20260429 14:08:38.167024 17918 sys_catalog.cc:458] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [sys.catalog]: This master's current role is: LEADER
I20260429 14:08:38.166925 17916 sys_catalog.cc:455] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af9649ac49224c7e91f0508bd22d69b4. Latest consensus state: current_term: 2 leader_uuid: "af9649ac49224c7e91f0508bd22d69b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9649ac49224c7e91f0508bd22d69b4" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 41969 } } }
I20260429 14:08:38.167214 17916 sys_catalog.cc:458] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [sys.catalog]: This master's current role is: LEADER
I20260429 14:08:38.167338 17921 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:08:38.168133 17921 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:08:38.168290 17921 catalog_manager.cc:1269] Loaded cluster ID: 92e01676e16043d284bee230b2910f0c
I20260429 14:08:38.168591 17921 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:08:38.168846 17921 catalog_manager.cc:1324] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260429 14:08:38.171586 17921 catalog_manager.cc:1334] IPKI private key successfully encrypted.
I20260429 14:08:38.174136 17921 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:08:38.174403 17921 catalog_manager.cc:6055] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4: Loaded TSK: 0
I20260429 14:08:38.174598 17921 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260429 14:08:38.238752 17914 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:08:38.238988 17914 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:08:38.239033 17914 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:08:38.240746 17914 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:08:38.240833 17914 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:08:38.240886 17914 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.193
I20260429 14:08:38.242753 17914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:43751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.193:34871
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.197.193
--webserver_port=43689
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:41969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.193
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:08:38.243039 17914 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:08:38.243281 17914 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260429 14:08:38.246243 17914 server_base.cc:1061] running on GCE node
W20260429 14:08:38.246461 17940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:38.246519 17942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:38.246678 17939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:08:38.246901 17914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:08:38.247166 17914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:08:38.248365 17914 hybrid_clock.cc:648] HybridClock initialized: now 1777471718248350 us; error 29 us; skew 500 ppm
Apr 29 14:08:38 dist-test-slave-8n77 krb5kdc[17280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471718, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:08:38.251125 17914 init.cc:377] Logged in from keytab as kudu/127.2.197.193@KRBTEST.COM (short username kudu)
I20260429 14:08:38.251797 17914 webserver.cc:492] Webserver started at http://127.2.197.193:43689/ using document root <none> and password file <none>
I20260429 14:08:38.251987 17914 fs_manager.cc:362] Metadata directory not provided
I20260429 14:08:38.252033 17914 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:08:38.253412 17914 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:08:38.254524 17949 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:38.254745 17914 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:08:38.254818 17914 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/wal
uuid: "cde63f442d164697b92f79a9c82b955a"
format_stamp: "Formatted at 2026-04-29 14:08:37 on dist-test-slave-8n77"
server_key: "38689aa1c0fb4b189491af6bab1b4580"
server_key_iv: "ff49f67e55bef8208152272e9b77cdcd"
server_key_version: "encryptionkey@0"
I20260429 14:08:38.255139 17914 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:08:38.283221 17914 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:08:38.285449 17914 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:08:38.285624 17914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:08:38.285914 17914 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:08:38.286515 17956 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260429 14:08:38.287633 17914 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260429 14:08:38.287722 17914 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260429 14:08:38.287778 17914 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260429 14:08:38.288374 17914 ts_tablet_manager.cc:616] Registered 1 tablets
I20260429 14:08:38.288441 17914 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260429 14:08:38.288463 17956 tablet_bootstrap.cc:492] T 06842361b98a438980f0d442eb328e2d P cde63f442d164697b92f79a9c82b955a: Bootstrap starting.
I20260429 14:08:38.296252 17914 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.193:34871
I20260429 14:08:38.296644 17914 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
I20260429 14:08:38.297261 18064 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.193:34871 every 8 connection(s)
I20260429 14:08:38.300467 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 17914
I20260429 14:08:38.300712 17956 log.cc:826] T 06842361b98a438980f0d442eb328e2d P cde63f442d164697b92f79a9c82b955a: Log is configured to *not* fsync() on all Append() calls
I20260429 14:08:38.300833 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.194:40909
--local_ip_for_outbound_sockets=127.2.197.194
--tserver_master_addrs=127.2.197.254:41969
--webserver_port=39345
--webserver_interface=127.2.197.194
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:43751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260429 14:08:38.303071 17956 tablet_bootstrap.cc:492] T 06842361b98a438980f0d442eb328e2d P cde63f442d164697b92f79a9c82b955a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260429 14:08:38.303388 17956 tablet_bootstrap.cc:492] T 06842361b98a438980f0d442eb328e2d P cde63f442d164697b92f79a9c82b955a: Bootstrap complete.
I20260429 14:08:38.304577 17956 ts_tablet_manager.cc:1403] T 06842361b98a438980f0d442eb328e2d P cde63f442d164697b92f79a9c82b955a: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.004s
I20260429 14:08:38.305704 17956 raft_consensus.cc:359] T 06842361b98a438980f0d442eb328e2d P cde63f442d164697b92f79a9c82b955a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcffd137fdc04dcfbf055e9c4529b211" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36003 } } peers { permanent_uuid: "7ecaaa30057f485f90fc8781084c9ef2" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40909 } } peers { permanent_uuid: "cde63f442d164697b92f79a9c82b955a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34871 } }
I20260429 14:08:38.305969 17956 raft_consensus.cc:740] T 06842361b98a438980f0d442eb328e2d P cde63f442d164697b92f79a9c82b955a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cde63f442d164697b92f79a9c82b955a, State: Initialized, Role: FOLLOWER
I20260429 14:08:38.306109 17956 consensus_queue.cc:260] T 06842361b98a438980f0d442eb328e2d P cde63f442d164697b92f79a9c82b955a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcffd137fdc04dcfbf055e9c4529b211" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36003 } } peers { permanent_uuid: "7ecaaa30057f485f90fc8781084c9ef2" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40909 } } peers { permanent_uuid: "cde63f442d164697b92f79a9c82b955a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34871 } }
I20260429 14:08:38.306411 17956 ts_tablet_manager.cc:1434] T 06842361b98a438980f0d442eb328e2d P cde63f442d164697b92f79a9c82b955a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
Apr 29 14:08:38 dist-test-slave-8n77 krb5kdc[17280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471718, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:08:38.313726 18068 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:38.297484 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.193:40401)
0429 14:08:38.300784 (+ 3300us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:38.301109 (+ 325us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:38.301930 (+ 821us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:38.302592 (+ 662us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:38.302597 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:38.303063 (+ 466us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:08:38.303430 (+ 367us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:38.303434 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:38.305170 (+ 1736us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:38.305171 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:38.305907 (+ 736us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:38.305909 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:38.306770 (+ 861us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:38.307484 (+ 714us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:08:38.307499 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:08:38.309438 (+ 1939us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:08:38.312066 (+ 2628us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:38.312074 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:38.312077 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:38.312365 (+ 288us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:38.312624 (+ 259us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:38.312624 (+ 0us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:38.312625 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:38.312711 (+ 86us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:38.313110 (+ 399us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:08:38.313115 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:08:38.313253 (+ 138us) client_negotiation.cc:770] Sending connection context
0429 14:08:38.313373 (+ 120us) client_negotiation.cc:241] Negotiation successful
0429 14:08:38.313538 (+ 165us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3253,"thread_start_us":188,"threads_started":1}
I20260429 14:08:38.314147 18067 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:38.297558 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:40401 (local address 127.2.197.254:41969)
0429 14:08:38.297747 (+ 189us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:38.297757 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:38.301249 (+ 3492us) server_negotiation.cc:408] Connection header received
0429 14:08:38.302142 (+ 893us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:38.302150 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:38.302207 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:38.302337 (+ 130us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:38.303628 (+ 1291us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:38.305048 (+ 1420us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:38.306272 (+ 1224us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:38.306642 (+ 370us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:38.309572 (+ 2930us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:38.309584 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:38.309584 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:38.309602 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:38.311945 (+ 2343us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:38.312476 (+ 531us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:38.312479 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:38.312479 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:38.312542 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:38.312789 (+ 247us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:38.312790 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:38.312790 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:38.313028 (+ 238us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:38.313122 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:38.313460 (+ 338us) server_negotiation.cc:300] Negotiation successful
0429 14:08:38.313609 (+ 149us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":120,"threads_started":1}
I20260429 14:08:38.314873 18065 heartbeater.cc:344] Connected to a master server at 127.2.197.254:41969
I20260429 14:08:38.314972 18065 heartbeater.cc:461] Registering TS with master...
I20260429 14:08:38.315191 18065 heartbeater.cc:507] Master 127.2.197.254:41969 requested a full tablet report, sending...
I20260429 14:08:38.315953 17876 ts_manager.cc:194] Registered new tserver with Master: cde63f442d164697b92f79a9c82b955a (127.2.197.193:34871)
I20260429 14:08:38.317175 17876 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.193@KRBTEST.COM'} at 127.2.197.193:40401
I20260429 14:08:38.318068 18065 heartbeater.cc:499] Master 127.2.197.254:41969 was elected leader, sending a full tablet report...
I20260429 14:08:38.321811 18072 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:38.316672 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:34871 (local address 127.0.0.1:48670)
0429 14:08:38.318391 (+ 1719us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:38.318422 (+ 31us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:38.318498 (+ 76us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:38.318773 (+ 275us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:38.318775 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:38.318778 (+ 3us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:08:38.319072 (+ 294us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:38.319074 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:38.320368 (+ 1294us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:38.320369 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:38.321312 (+ 943us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:38.321316 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:38.321602 (+ 286us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:38.321628 (+ 26us) client_negotiation.cc:770] Sending connection context
0429 14:08:38.321681 (+ 53us) client_negotiation.cc:241] Negotiation successful
0429 14:08:38.321727 (+ 46us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1682,"thread_start_us":104,"threads_started":1}
I20260429 14:08:38.322263 18073 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:38.316918 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48670 (local address 127.2.197.193:34871)
0429 14:08:38.318617 (+ 1699us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:38.318624 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:38.318633 (+ 9us) server_negotiation.cc:408] Connection header received
0429 14:08:38.318666 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:38.318668 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:38.318683 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:38.318747 (+ 64us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:08:38.319161 (+ 414us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:38.320266 (+ 1105us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:38.321418 (+ 1152us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:38.321951 (+ 533us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:38.321991 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:38.322023 (+ 32us) server_negotiation.cc:300] Negotiation successful
0429 14:08:38.322060 (+ 37us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1671,"thread_start_us":66,"threads_started":1}
I20260429 14:08:38.322770 17998 tablet_service.cc:1558] Processing DeleteTablet for tablet 06842361b98a438980f0d442eb328e2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:08:37 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:48670
I20260429 14:08:38.323755 18074 tablet_replica.cc:333] stopping tablet replica
I20260429 14:08:38.323844 18074 raft_consensus.cc:2243] T 06842361b98a438980f0d442eb328e2d P cde63f442d164697b92f79a9c82b955a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:08:38.323910 18074 raft_consensus.cc:2272] T 06842361b98a438980f0d442eb328e2d P cde63f442d164697b92f79a9c82b955a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:08:38.324244 18074 ts_tablet_manager.cc:1916] T 06842361b98a438980f0d442eb328e2d P cde63f442d164697b92f79a9c82b955a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:08:38.326517 18074 ts_tablet_manager.cc:1929] T 06842361b98a438980f0d442eb328e2d P cde63f442d164697b92f79a9c82b955a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 14:08:38.326597 18074 log.cc:1199] T 06842361b98a438980f0d442eb328e2d P cde63f442d164697b92f79a9c82b955a: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/wal/wals/06842361b98a438980f0d442eb328e2d
I20260429 14:08:38.326845 18074 ts_tablet_manager.cc:1950] T 06842361b98a438980f0d442eb328e2d P cde63f442d164697b92f79a9c82b955a: Deleting consensus metadata
I20260429 14:08:38.327397 17861 catalog_manager.cc:5002] TS cde63f442d164697b92f79a9c82b955a (127.2.197.193:34871): tablet 06842361b98a438980f0d442eb328e2d (table test-table [id=512ea896dd6848ae9cdece8f6f054f59]) successfully deleted
W20260429 14:08:38.388697 18070 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:08:38.388896 18070 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:08:38.388921 18070 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:08:38.390432 18070 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:08:38.390486 18070 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:08:38.390515 18070 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.194
I20260429 14:08:38.392154 18070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:43751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.194:40909
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.197.194
--webserver_port=39345
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:41969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.194
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:08:38.392383 18070 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:08:38.392737 18070 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:08:38.395844 18082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:38.395838 18080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:08:38.395874 18070 server_base.cc:1061] running on GCE node
W20260429 14:08:38.395862 18079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:08:38.396360 18070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:08:38.396651 18070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:08:38.397828 18070 hybrid_clock.cc:648] HybridClock initialized: now 1777471718397817 us; error 52 us; skew 500 ppm
Apr 29 14:08:38 dist-test-slave-8n77 krb5kdc[17280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471718, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:08:38.400779 18070 init.cc:377] Logged in from keytab as kudu/127.2.197.194@KRBTEST.COM (short username kudu)
I20260429 14:08:38.401407 18070 webserver.cc:492] Webserver started at http://127.2.197.194:39345/ using document root <none> and password file <none>
I20260429 14:08:38.401629 18070 fs_manager.cc:362] Metadata directory not provided
I20260429 14:08:38.401683 18070 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:08:38.403008 18070 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260429 14:08:38.403648 18089 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:38.403965 18070 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:08:38.404042 18070 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/wal
uuid: "7ecaaa30057f485f90fc8781084c9ef2"
format_stamp: "Formatted at 2026-04-29 14:08:37 on dist-test-slave-8n77"
server_key: "1873e40a0ca76054160f00f6125fd435"
server_key_iv: "60ad4af3288d8fb69c32ab12bab2d918"
server_key_version: "encryptionkey@0"
I20260429 14:08:38.404341 18070 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:08:38.419442 18070 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:08:38.421479 18070 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:08:38.421664 18070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:08:38.421908 18070 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:08:38.422224 18070 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:08:38.422257 18070 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:38.422362 18070 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:08:38.422384 18070 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:38.428253 18070 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.194:40909
I20260429 14:08:38.428290 18202 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.194:40909 every 8 connection(s)
I20260429 14:08:38.428625 18070 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
Apr 29 14:08:38 dist-test-slave-8n77 krb5kdc[17280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471718, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:08:38.437332 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 18070
I20260429 14:08:38.437780 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.195:36003
--local_ip_for_outbound_sockets=127.2.197.195
--tserver_master_addrs=127.2.197.254:41969
--webserver_port=38219
--webserver_interface=127.2.197.195
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:43751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260429 14:08:38.438174 18067 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:38.429510 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:60825 (local address 127.2.197.254:41969)
0429 14:08:38.429622 (+ 112us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:38.429625 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:38.429793 (+ 168us) server_negotiation.cc:408] Connection header received
0429 14:08:38.430328 (+ 535us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:38.430329 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:38.430347 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:38.430406 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:38.431140 (+ 734us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:38.431715 (+ 575us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:38.432300 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:38.432477 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:38.435312 (+ 2835us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:38.435318 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:38.435319 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:38.435329 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:38.436703 (+ 1374us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:38.437154 (+ 451us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:38.437156 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:38.437156 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:38.437190 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:38.437410 (+ 220us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:38.437411 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:38.437412 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:38.437523 (+ 111us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:38.437653 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:38.437998 (+ 345us) server_negotiation.cc:300] Negotiation successful
0429 14:08:38.438102 (+ 104us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260429 14:08:38.438627 18205 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:38.429399 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.194:60825)
0429 14:08:38.429665 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:38.429704 (+ 39us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:38.430200 (+ 496us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:38.430482 (+ 282us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:38.430486 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:38.430788 (+ 302us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:08:38.431033 (+ 245us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:38.431036 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:38.431812 (+ 776us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:38.431813 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:38.432212 (+ 399us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:38.432214 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:38.432348 (+ 134us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:38.432958 (+ 610us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:08:38.432967 (+ 9us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:08:38.435216 (+ 2249us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:08:38.436828 (+ 1612us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:38.436835 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:38.436839 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:38.437058 (+ 219us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:38.437258 (+ 200us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:38.437259 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:38.437259 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:38.437341 (+ 82us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:38.437643 (+ 302us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:08:38.437648 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:08:38.437797 (+ 149us) client_negotiation.cc:770] Sending connection context
0429 14:08:38.438329 (+ 532us) client_negotiation.cc:241] Negotiation successful
0429 14:08:38.438469 (+ 140us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":106,"threads_started":1}
I20260429 14:08:38.439225 18203 heartbeater.cc:344] Connected to a master server at 127.2.197.254:41969
I20260429 14:08:38.439347 18203 heartbeater.cc:461] Registering TS with master...
I20260429 14:08:38.439639 18203 heartbeater.cc:507] Master 127.2.197.254:41969 requested a full tablet report, sending...
I20260429 14:08:38.440099 17876 ts_manager.cc:194] Registered new tserver with Master: 7ecaaa30057f485f90fc8781084c9ef2 (127.2.197.194:40909)
I20260429 14:08:38.440506 17876 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.194@KRBTEST.COM'} at 127.2.197.194:60825
W20260429 14:08:38.518702 18206 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:08:38.518923 18206 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:08:38.518954 18206 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:08:38.520597 18206 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:08:38.520673 18206 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:08:38.520706 18206 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.195
I20260429 14:08:38.522548 18206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:43751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.195:36003
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.197.195
--webserver_port=38219
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:41969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.195
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:08:38.522820 18206 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:08:38.523037 18206 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:08:38.525827 18214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:38.525822 18211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:38.525818 18212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:08:38.526198 18206 server_base.cc:1061] running on GCE node
I20260429 14:08:38.526432 18206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:08:38.526726 18206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:08:38.527912 18206 hybrid_clock.cc:648] HybridClock initialized: now 1777471718527889 us; error 33 us; skew 500 ppm
Apr 29 14:08:38 dist-test-slave-8n77 krb5kdc[17280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471718, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:08:38.530543 18206 init.cc:377] Logged in from keytab as kudu/127.2.197.195@KRBTEST.COM (short username kudu)
I20260429 14:08:38.531157 18206 webserver.cc:492] Webserver started at http://127.2.197.195:38219/ using document root <none> and password file <none>
I20260429 14:08:38.531369 18206 fs_manager.cc:362] Metadata directory not provided
I20260429 14:08:38.531454 18206 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:08:38.532797 18206 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:08:38.533600 18221 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:38.533872 18206 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:08:38.533964 18206 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/wal
uuid: "bcffd137fdc04dcfbf055e9c4529b211"
format_stamp: "Formatted at 2026-04-29 14:08:37 on dist-test-slave-8n77"
server_key: "25e53cf0c3badb473a63e257bf278ae6"
server_key_iv: "6c0059b5d7a1c3086c5fb24f1b46f208"
server_key_version: "encryptionkey@0"
I20260429 14:08:38.534416 18206 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:08:38.548645 18206 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:08:38.550747 18206 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:08:38.550927 18206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:08:38.551167 18206 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:08:38.551483 18206 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:08:38.551538 18206 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:38.551580 18206 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:08:38.551610 18206 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:38.557765 18206 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.195:36003
I20260429 14:08:38.557807 18334 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.195:36003 every 8 connection(s)
I20260429 14:08:38.558106 18206 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
Apr 29 14:08:38 dist-test-slave-8n77 krb5kdc[17280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471718, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:08:38.563938 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 18206
I20260429 14:08:38.567494 18067 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:38.558853 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:32877 (local address 127.2.197.254:41969)
0429 14:08:38.558919 (+ 66us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:38.558921 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:38.559187 (+ 266us) server_negotiation.cc:408] Connection header received
0429 14:08:38.559703 (+ 516us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:38.559704 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:38.559717 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:38.559773 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:38.560522 (+ 749us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:38.561049 (+ 527us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:38.561624 (+ 575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:38.561799 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:38.564466 (+ 2667us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:38.564470 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:38.564471 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:38.564479 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:38.566044 (+ 1565us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:38.566519 (+ 475us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:38.566520 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:38.566521 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:38.566548 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:38.566763 (+ 215us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:38.566764 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:38.566765 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:38.566854 (+ 89us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:38.566928 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:38.567243 (+ 315us) server_negotiation.cc:300] Negotiation successful
0429 14:08:38.567385 (+ 142us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260429 14:08:38.567567 18337 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:38.558896 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.195:32877)
0429 14:08:38.559060 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:38.559089 (+ 29us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:38.559598 (+ 509us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:38.559826 (+ 228us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:38.559830 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:38.560126 (+ 296us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:08:38.560418 (+ 292us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:38.560421 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:38.561147 (+ 726us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:38.561148 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:38.561533 (+ 385us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:38.561534 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:38.561663 (+ 129us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:38.562183 (+ 520us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:08:38.562193 (+ 10us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:08:38.564329 (+ 2136us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:08:38.566131 (+ 1802us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:38.566136 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:38.566139 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:38.566429 (+ 290us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:38.566621 (+ 192us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:38.566622 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:38.566622 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:38.566698 (+ 76us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:38.566917 (+ 219us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:08:38.566923 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:08:38.567055 (+ 132us) client_negotiation.cc:770] Sending connection context
0429 14:08:38.567203 (+ 148us) client_negotiation.cc:241] Negotiation successful
0429 14:08:38.567361 (+ 158us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":66,"threads_started":1}
I20260429 14:08:38.568066 18335 heartbeater.cc:344] Connected to a master server at 127.2.197.254:41969
I20260429 14:08:38.568166 18335 heartbeater.cc:461] Registering TS with master...
I20260429 14:08:38.568398 18335 heartbeater.cc:507] Master 127.2.197.254:41969 requested a full tablet report, sending...
I20260429 14:08:38.568845 17876 ts_manager.cc:194] Registered new tserver with Master: bcffd137fdc04dcfbf055e9c4529b211 (127.2.197.195:36003)
I20260429 14:08:38.569226 17876 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.195@KRBTEST.COM'} at 127.2.197.195:32877
I20260429 14:08:38.571166 18339 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:38.564393 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51482 (local address 127.2.197.254:41969)
0429 14:08:38.564679 (+ 286us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:38.564681 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:38.564690 (+ 9us) server_negotiation.cc:408] Connection header received
0429 14:08:38.564711 (+ 21us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:38.564712 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:38.564724 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:38.564786 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:38.565746 (+ 960us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:38.566375 (+ 629us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:38.566952 (+ 577us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:38.567143 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:38.568324 (+ 1181us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:38.568327 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:38.568327 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:38.568352 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:38.570068 (+ 1716us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:38.570474 (+ 406us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:38.570476 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:38.570476 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:38.570499 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:38.570663 (+ 164us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:38.570664 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:38.570664 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:38.570758 (+ 94us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:38.570924 (+ 166us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:38.570969 (+ 45us) server_negotiation.cc:300] Negotiation successful
0429 14:08:38.571057 (+ 88us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":79,"threads_started":1}
I20260429 14:08:38.572232 2839 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260429 14:08:38.575169 18339 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:38.572786 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51498 (local address 127.2.197.254:41969)
0429 14:08:38.572871 (+ 85us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:38.572873 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:38.572993 (+ 120us) server_negotiation.cc:408] Connection header received
0429 14:08:38.573105 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:38.573106 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:38.573133 (+ 27us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:38.573210 (+ 77us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:08:38.573538 (+ 328us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:38.574037 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:38.574657 (+ 620us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:38.574807 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:38.574837 (+ 30us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:08:38.574918 (+ 81us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:08:38.575004 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:38.575062 (+ 58us) server_negotiation.cc:300] Negotiation successful
0429 14:08:38.575090 (+ 28us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260429 14:08:38.575551 17876 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:51498:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 14:08:38.576821 17876 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 14:08:38.579154 17998 tablet_service.cc:1511] Processing CreateTablet for tablet 2d87ab35f9f343e08090b48099a2e21d (DEFAULT_TABLE table=test-table [id=6c99f8ea9b2945f7901a6e8eab90456c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:08:38.579357 17998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d87ab35f9f343e08090b48099a2e21d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:08:38.580892 17956 tablet_bootstrap.cc:492] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a: Bootstrap starting.
I20260429 14:08:38.581305 17956 tablet_bootstrap.cc:654] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a: Neither blocks nor log segments found. Creating new log.
I20260429 14:08:38.581707 18341 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:38.578727 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:36003 (local address 127.0.0.1:42702)
0429 14:08:38.578976 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:38.578984 (+ 8us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:38.579027 (+ 43us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:38.579245 (+ 218us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:38.579245 (+ 0us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:38.579248 (+ 3us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:08:38.579588 (+ 340us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:38.579591 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:38.580549 (+ 958us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:38.580550 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:38.581453 (+ 903us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:38.581456 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:38.581540 (+ 84us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:38.581546 (+ 6us) client_negotiation.cc:770] Sending connection context
0429 14:08:38.581593 (+ 47us) client_negotiation.cc:241] Negotiation successful
0429 14:08:38.581635 (+ 42us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":28,"thread_start_us":106,"threads_started":1}
I20260429 14:08:38.582259 18342 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:38.578848 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42702 (local address 127.2.197.195:36003)
0429 14:08:38.579090 (+ 242us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:38.579097 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:38.579106 (+ 9us) server_negotiation.cc:408] Connection header received
0429 14:08:38.579150 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:38.579151 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:38.579164 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:38.579243 (+ 79us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:08:38.579674 (+ 431us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:38.580460 (+ 786us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:38.581558 (+ 1098us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:38.582098 (+ 540us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:38.582139 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:38.582166 (+ 27us) server_negotiation.cc:300] Negotiation successful
0429 14:08:38.582194 (+ 28us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":73,"threads_started":1}
I20260429 14:08:38.582536 18072 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:38.578727 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:40909 (local address 127.0.0.1:35838)
0429 14:08:38.578787 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:38.578799 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:38.578840 (+ 41us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:38.579378 (+ 538us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:38.579379 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:38.579383 (+ 4us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:08:38.579564 (+ 181us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:38.579565 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:38.581604 (+ 2039us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:38.581606 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:38.582243 (+ 637us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:38.582245 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:38.582369 (+ 124us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:38.582374 (+ 5us) client_negotiation.cc:770] Sending connection context
0429 14:08:38.582421 (+ 47us) client_negotiation.cc:241] Negotiation successful
0429 14:08:38.582462 (+ 41us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260429 14:08:38.582988 18269 tablet_service.cc:1511] Processing CreateTablet for tablet 2d87ab35f9f343e08090b48099a2e21d (DEFAULT_TABLE table=test-table [id=6c99f8ea9b2945f7901a6e8eab90456c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:08:38.583271 18269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d87ab35f9f343e08090b48099a2e21d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:08:38.583740 18343 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:38.578976 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35838 (local address 127.2.197.194:40909)
0429 14:08:38.579150 (+ 174us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:38.579157 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:38.579169 (+ 12us) server_negotiation.cc:408] Connection header received
0429 14:08:38.579215 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:38.579217 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:38.579235 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:38.579355 (+ 120us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:08:38.579652 (+ 297us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:38.580796 (+ 1144us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:38.582423 (+ 1627us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:38.583534 (+ 1111us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:38.583579 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:38.583611 (+ 32us) server_negotiation.cc:300] Negotiation successful
0429 14:08:38.583667 (+ 56us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":97,"threads_started":1}
I20260429 14:08:38.584259 18137 tablet_service.cc:1511] Processing CreateTablet for tablet 2d87ab35f9f343e08090b48099a2e21d (DEFAULT_TABLE table=test-table [id=6c99f8ea9b2945f7901a6e8eab90456c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:08:38.584549 18137 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d87ab35f9f343e08090b48099a2e21d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:08:38.585742 18345 tablet_bootstrap.cc:492] T 2d87ab35f9f343e08090b48099a2e21d P bcffd137fdc04dcfbf055e9c4529b211: Bootstrap starting.
I20260429 14:08:38.586022 17956 tablet_bootstrap.cc:492] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a: No bootstrap required, opened a new log
I20260429 14:08:38.586097 17956 ts_tablet_manager.cc:1403] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260429 14:08:38.586226 17956 raft_consensus.cc:359] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcffd137fdc04dcfbf055e9c4529b211" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36003 } } peers { permanent_uuid: "7ecaaa30057f485f90fc8781084c9ef2" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40909 } } peers { permanent_uuid: "cde63f442d164697b92f79a9c82b955a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34871 } }
I20260429 14:08:38.586331 17956 raft_consensus.cc:385] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:08:38.586400 17956 raft_consensus.cc:740] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cde63f442d164697b92f79a9c82b955a, State: Initialized, Role: FOLLOWER
I20260429 14:08:38.586428 18345 tablet_bootstrap.cc:654] T 2d87ab35f9f343e08090b48099a2e21d P bcffd137fdc04dcfbf055e9c4529b211: Neither blocks nor log segments found. Creating new log.
I20260429 14:08:38.586457 17956 consensus_queue.cc:260] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcffd137fdc04dcfbf055e9c4529b211" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36003 } } peers { permanent_uuid: "7ecaaa30057f485f90fc8781084c9ef2" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40909 } } peers { permanent_uuid: "cde63f442d164697b92f79a9c82b955a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34871 } }
I20260429 14:08:38.586567 17956 ts_tablet_manager.cc:1434] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260429 14:08:38.586730 18345 log.cc:826] T 2d87ab35f9f343e08090b48099a2e21d P bcffd137fdc04dcfbf055e9c4529b211: Log is configured to *not* fsync() on all Append() calls
I20260429 14:08:38.587373 18347 tablet_bootstrap.cc:492] T 2d87ab35f9f343e08090b48099a2e21d P 7ecaaa30057f485f90fc8781084c9ef2: Bootstrap starting.
I20260429 14:08:38.587514 18345 tablet_bootstrap.cc:492] T 2d87ab35f9f343e08090b48099a2e21d P bcffd137fdc04dcfbf055e9c4529b211: No bootstrap required, opened a new log
I20260429 14:08:38.587566 18345 ts_tablet_manager.cc:1403] T 2d87ab35f9f343e08090b48099a2e21d P bcffd137fdc04dcfbf055e9c4529b211: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260429 14:08:38.587901 18347 tablet_bootstrap.cc:654] T 2d87ab35f9f343e08090b48099a2e21d P 7ecaaa30057f485f90fc8781084c9ef2: Neither blocks nor log segments found. Creating new log.
I20260429 14:08:38.588186 18347 log.cc:826] T 2d87ab35f9f343e08090b48099a2e21d P 7ecaaa30057f485f90fc8781084c9ef2: Log is configured to *not* fsync() on all Append() calls
I20260429 14:08:38.588670 18345 raft_consensus.cc:359] T 2d87ab35f9f343e08090b48099a2e21d P bcffd137fdc04dcfbf055e9c4529b211 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcffd137fdc04dcfbf055e9c4529b211" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36003 } } peers { permanent_uuid: "7ecaaa30057f485f90fc8781084c9ef2" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40909 } } peers { permanent_uuid: "cde63f442d164697b92f79a9c82b955a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34871 } }
I20260429 14:08:38.588845 18345 raft_consensus.cc:385] T 2d87ab35f9f343e08090b48099a2e21d P bcffd137fdc04dcfbf055e9c4529b211 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:08:38.588881 18345 raft_consensus.cc:740] T 2d87ab35f9f343e08090b48099a2e21d P bcffd137fdc04dcfbf055e9c4529b211 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcffd137fdc04dcfbf055e9c4529b211, State: Initialized, Role: FOLLOWER
I20260429 14:08:38.589073 18347 tablet_bootstrap.cc:492] T 2d87ab35f9f343e08090b48099a2e21d P 7ecaaa30057f485f90fc8781084c9ef2: No bootstrap required, opened a new log
I20260429 14:08:38.589148 18347 ts_tablet_manager.cc:1403] T 2d87ab35f9f343e08090b48099a2e21d P 7ecaaa30057f485f90fc8781084c9ef2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260429 14:08:38.588990 18345 consensus_queue.cc:260] T 2d87ab35f9f343e08090b48099a2e21d P bcffd137fdc04dcfbf055e9c4529b211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcffd137fdc04dcfbf055e9c4529b211" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36003 } } peers { permanent_uuid: "7ecaaa30057f485f90fc8781084c9ef2" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40909 } } peers { permanent_uuid: "cde63f442d164697b92f79a9c82b955a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34871 } }
I20260429 14:08:38.589507 18345 ts_tablet_manager.cc:1434] T 2d87ab35f9f343e08090b48099a2e21d P bcffd137fdc04dcfbf055e9c4529b211: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260429 14:08:38.589636 18335 heartbeater.cc:499] Master 127.2.197.254:41969 was elected leader, sending a full tablet report...
I20260429 14:08:38.590788 18347 raft_consensus.cc:359] T 2d87ab35f9f343e08090b48099a2e21d P 7ecaaa30057f485f90fc8781084c9ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcffd137fdc04dcfbf055e9c4529b211" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36003 } } peers { permanent_uuid: "7ecaaa30057f485f90fc8781084c9ef2" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40909 } } peers { permanent_uuid: "cde63f442d164697b92f79a9c82b955a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34871 } }
I20260429 14:08:38.590894 18347 raft_consensus.cc:385] T 2d87ab35f9f343e08090b48099a2e21d P 7ecaaa30057f485f90fc8781084c9ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:08:38.590953 18347 raft_consensus.cc:740] T 2d87ab35f9f343e08090b48099a2e21d P 7ecaaa30057f485f90fc8781084c9ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ecaaa30057f485f90fc8781084c9ef2, State: Initialized, Role: FOLLOWER
I20260429 14:08:38.591068 18347 consensus_queue.cc:260] T 2d87ab35f9f343e08090b48099a2e21d P 7ecaaa30057f485f90fc8781084c9ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcffd137fdc04dcfbf055e9c4529b211" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36003 } } peers { permanent_uuid: "7ecaaa30057f485f90fc8781084c9ef2" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40909 } } peers { permanent_uuid: "cde63f442d164697b92f79a9c82b955a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34871 } }
I20260429 14:08:38.591334 18347 ts_tablet_manager.cc:1434] T 2d87ab35f9f343e08090b48099a2e21d P 7ecaaa30057f485f90fc8781084c9ef2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260429 14:08:38.591369 18203 heartbeater.cc:499] Master 127.2.197.254:41969 was elected leader, sending a full tablet report...
W20260429 14:08:38.679246 18204 tablet.cc:2404] T 2d87ab35f9f343e08090b48099a2e21d P 7ecaaa30057f485f90fc8781084c9ef2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 14:08:38.746485 18349 raft_consensus.cc:493] T 2d87ab35f9f343e08090b48099a2e21d P bcffd137fdc04dcfbf055e9c4529b211 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:08:38.746645 18349 raft_consensus.cc:515] T 2d87ab35f9f343e08090b48099a2e21d P bcffd137fdc04dcfbf055e9c4529b211 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcffd137fdc04dcfbf055e9c4529b211" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36003 } } peers { permanent_uuid: "7ecaaa30057f485f90fc8781084c9ef2" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40909 } } peers { permanent_uuid: "cde63f442d164697b92f79a9c82b955a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34871 } }
I20260429 14:08:38.747015 18349 leader_election.cc:290] T 2d87ab35f9f343e08090b48099a2e21d P bcffd137fdc04dcfbf055e9c4529b211 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ecaaa30057f485f90fc8781084c9ef2 (127.2.197.194:40909), cde63f442d164697b92f79a9c82b955a (127.2.197.193:34871)
I20260429 14:08:38.749725 18337 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:38.747290 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:34871 (local address 127.2.197.195:48297)
0429 14:08:38.747502 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:38.747516 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:38.747572 (+ 56us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:38.747836 (+ 264us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:38.747837 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:38.747839 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:08:38.748010 (+ 171us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:38.748012 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:38.748707 (+ 695us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:38.748708 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:38.749546 (+ 838us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:38.749548 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:38.749615 (+ 67us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:38.749618 (+ 3us) client_negotiation.cc:770] Sending connection context
0429 14:08:38.749636 (+ 18us) client_negotiation.cc:241] Negotiation successful
0429 14:08:38.749667 (+ 31us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":12,"thread_start_us":148,"threads_started":1}
I20260429 14:08:38.750234 18073 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:38.747414 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:48297 (local address 127.2.197.193:34871)
0429 14:08:38.747467 (+ 53us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:38.747470 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:38.747562 (+ 92us) server_negotiation.cc:408] Connection header received
0429 14:08:38.747703 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:38.747704 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:38.747716 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:38.747773 (+ 57us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:08:38.748093 (+ 320us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:38.748628 (+ 535us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:38.749660 (+ 1032us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:38.750091 (+ 431us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:38.750100 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:38.750122 (+ 22us) server_negotiation.cc:300] Negotiation successful
0429 14:08:38.750158 (+ 36us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260429 14:08:38.750353 18351 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:38.747279 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:40909 (local address 127.2.197.195:43617)
0429 14:08:38.747502 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:38.747515 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:38.747572 (+ 57us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:38.747802 (+ 230us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:38.747803 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:38.747806 (+ 3us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:08:38.748120 (+ 314us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:38.748130 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:38.749282 (+ 1152us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:38.749283 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:38.750102 (+ 819us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:38.750105 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:38.750183 (+ 78us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:38.750187 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 14:08:38.750210 (+ 23us) client_negotiation.cc:241] Negotiation successful
0429 14:08:38.750243 (+ 33us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201}
I20260429 14:08:38.750552 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d87ab35f9f343e08090b48099a2e21d" candidate_uuid: "bcffd137fdc04dcfbf055e9c4529b211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cde63f442d164697b92f79a9c82b955a" is_pre_election: true
I20260429 14:08:38.750715 18019 raft_consensus.cc:2468] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcffd137fdc04dcfbf055e9c4529b211 in term 0.
I20260429 14:08:38.750962 18223 leader_election.cc:304] T 2d87ab35f9f343e08090b48099a2e21d P bcffd137fdc04dcfbf055e9c4529b211 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcffd137fdc04dcfbf055e9c4529b211, cde63f442d164697b92f79a9c82b955a; no voters:
I20260429 14:08:38.751101 18349 raft_consensus.cc:2804] T 2d87ab35f9f343e08090b48099a2e21d P bcffd137fdc04dcfbf055e9c4529b211 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 14:08:38.751129 18343 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:38.747501 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:43617 (local address 127.2.197.194:40909)
0429 14:08:38.747544 (+ 43us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:38.747546 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:38.747556 (+ 10us) server_negotiation.cc:408] Connection header received
0429 14:08:38.747690 (+ 134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:38.747692 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:38.747742 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:38.747796 (+ 54us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:08:38.748733 (+ 937us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:38.749176 (+ 443us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:38.750258 (+ 1082us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:38.750956 (+ 698us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:38.750965 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:38.750988 (+ 23us) server_negotiation.cc:300] Negotiation successful
0429 14:08:38.751016 (+ 28us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260429 14:08:38.751157 18349 raft_consensus.cc:493] T 2d87ab35f9f343e08090b48099a2e21d P bcffd137fdc04dcfbf055e9c4529b211 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 14:08:38.751176 18349 raft_consensus.cc:3060] T 2d87ab35f9f343e08090b48099a2e21d P bcffd137fdc04dcfbf055e9c4529b211 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:08:38.751441 18157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d87ab35f9f343e08090b48099a2e21d" candidate_uuid: "bcffd137fdc04dcfbf055e9c4529b211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ecaaa30057f485f90fc8781084c9ef2" is_pre_election: true
I20260429 14:08:38.751593 18157 raft_consensus.cc:2468] T 2d87ab35f9f343e08090b48099a2e21d P 7ecaaa30057f485f90fc8781084c9ef2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcffd137fdc04dcfbf055e9c4529b211 in term 0.
I20260429 14:08:38.751820 18349 raft_consensus.cc:515] T 2d87ab35f9f343e08090b48099a2e21d P bcffd137fdc04dcfbf055e9c4529b211 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcffd137fdc04dcfbf055e9c4529b211" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36003 } } peers { permanent_uuid: "7ecaaa30057f485f90fc8781084c9ef2" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40909 } } peers { permanent_uuid: "cde63f442d164697b92f79a9c82b955a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34871 } }
I20260429 14:08:38.751977 18349 leader_election.cc:290] T 2d87ab35f9f343e08090b48099a2e21d P bcffd137fdc04dcfbf055e9c4529b211 [CANDIDATE]: Term 1 election: Requested vote from peers 7ecaaa30057f485f90fc8781084c9ef2 (127.2.197.194:40909), cde63f442d164697b92f79a9c82b955a (127.2.197.193:34871)
I20260429 14:08:38.752185 18157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d87ab35f9f343e08090b48099a2e21d" candidate_uuid: "bcffd137fdc04dcfbf055e9c4529b211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ecaaa30057f485f90fc8781084c9ef2"
I20260429 14:08:38.752269 18157 raft_consensus.cc:3060] T 2d87ab35f9f343e08090b48099a2e21d P 7ecaaa30057f485f90fc8781084c9ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:08:38.752955 18157 raft_consensus.cc:2468] T 2d87ab35f9f343e08090b48099a2e21d P 7ecaaa30057f485f90fc8781084c9ef2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcffd137fdc04dcfbf055e9c4529b211 in term 1.
I20260429 14:08:38.753062 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d87ab35f9f343e08090b48099a2e21d" candidate_uuid: "bcffd137fdc04dcfbf055e9c4529b211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cde63f442d164697b92f79a9c82b955a"
I20260429 14:08:38.753166 18019 raft_consensus.cc:3060] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:08:38.753189 18225 leader_election.cc:304] T 2d87ab35f9f343e08090b48099a2e21d P bcffd137fdc04dcfbf055e9c4529b211 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ecaaa30057f485f90fc8781084c9ef2, bcffd137fdc04dcfbf055e9c4529b211; no voters:
I20260429 14:08:38.753270 18349 raft_consensus.cc:2804] T 2d87ab35f9f343e08090b48099a2e21d P bcffd137fdc04dcfbf055e9c4529b211 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:08:38.753556 18349 raft_consensus.cc:697] T 2d87ab35f9f343e08090b48099a2e21d P bcffd137fdc04dcfbf055e9c4529b211 [term 1 LEADER]: Becoming Leader. State: Replica: bcffd137fdc04dcfbf055e9c4529b211, State: Running, Role: LEADER
I20260429 14:08:38.753749 18349 consensus_queue.cc:237] T 2d87ab35f9f343e08090b48099a2e21d P bcffd137fdc04dcfbf055e9c4529b211 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcffd137fdc04dcfbf055e9c4529b211" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36003 } } peers { permanent_uuid: "7ecaaa30057f485f90fc8781084c9ef2" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40909 } } peers { permanent_uuid: "cde63f442d164697b92f79a9c82b955a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34871 } }
I20260429 14:08:38.754127 18019 raft_consensus.cc:2468] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcffd137fdc04dcfbf055e9c4529b211 in term 1.
I20260429 14:08:38.754774 17875 catalog_manager.cc:5671] T 2d87ab35f9f343e08090b48099a2e21d P bcffd137fdc04dcfbf055e9c4529b211 reported cstate change: term changed from 0 to 1, leader changed from <none> to bcffd137fdc04dcfbf055e9c4529b211 (127.2.197.195). New cstate: current_term: 1 leader_uuid: "bcffd137fdc04dcfbf055e9c4529b211" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcffd137fdc04dcfbf055e9c4529b211" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ecaaa30057f485f90fc8781084c9ef2" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cde63f442d164697b92f79a9c82b955a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34871 } health_report { overall_health: UNKNOWN } } }
I20260429 14:08:38.769093 18342 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:38.766852 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42704 (local address 127.2.197.195:36003)
0429 14:08:38.766914 (+ 62us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:38.766916 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:38.766931 (+ 15us) server_negotiation.cc:408] Connection header received
0429 14:08:38.766970 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:38.766971 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:38.766984 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:38.767035 (+ 51us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:08:38.767351 (+ 316us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:38.767915 (+ 564us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:38.768538 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:38.768685 (+ 147us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:38.768719 (+ 34us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:08:38.768806 (+ 87us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:08:38.768873 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:38.768930 (+ 57us) server_negotiation.cc:300] Negotiation successful
0429 14:08:38.768955 (+ 25us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260429 14:08:38.770790 18157 raft_consensus.cc:1275] T 2d87ab35f9f343e08090b48099a2e21d P 7ecaaa30057f485f90fc8781084c9ef2 [term 1 FOLLOWER]: Refusing update from remote peer bcffd137fdc04dcfbf055e9c4529b211: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:08:38.770792 18019 raft_consensus.cc:1275] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [term 1 FOLLOWER]: Refusing update from remote peer bcffd137fdc04dcfbf055e9c4529b211: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:08:38.771184 18349 consensus_queue.cc:1048] T 2d87ab35f9f343e08090b48099a2e21d P bcffd137fdc04dcfbf055e9c4529b211 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cde63f442d164697b92f79a9c82b955a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:08:38.771325 18352 consensus_queue.cc:1048] T 2d87ab35f9f343e08090b48099a2e21d P bcffd137fdc04dcfbf055e9c4529b211 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ecaaa30057f485f90fc8781084c9ef2" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:08:38.774497 18356 mvcc.cc:204] Tried to move back new op lower bound from 7280524160082251776 to 7280524160016039936. Current Snapshot: MvccSnapshot[applied={T|T < 7280524160082251776 or (T in {7280524160082251776})}]
I20260429 14:08:38.775085 18355 mvcc.cc:204] Tried to move back new op lower bound from 7280524160082251776 to 7280524160016039936. Current Snapshot: MvccSnapshot[applied={T|T < 7280524160082251776 or (T in {7280524160082251776})}]
I20260429 14:08:38.776444 17876 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:51498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:08:38.776528 17876 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:51498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:08:38.777314 17876 catalog_manager.cc:5958] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4: Sending DeleteTablet for 3 replicas of tablet 2d87ab35f9f343e08090b48099a2e21d
I20260429 14:08:38.777681 17998 tablet_service.cc:1558] Processing DeleteTablet for tablet 2d87ab35f9f343e08090b48099a2e21d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:08:38 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:48670
I20260429 14:08:38.777659 18269 tablet_service.cc:1558] Processing DeleteTablet for tablet 2d87ab35f9f343e08090b48099a2e21d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:08:38 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:42702
I20260429 14:08:38.777791 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 17914
I20260429 14:08:38.777774 18137 tablet_service.cc:1558] Processing DeleteTablet for tablet 2d87ab35f9f343e08090b48099a2e21d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:08:38 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:35838
I20260429 14:08:38.778008 18365 tablet_replica.cc:333] T 2d87ab35f9f343e08090b48099a2e21d P 7ecaaa30057f485f90fc8781084c9ef2: stopping tablet replica
I20260429 14:08:38.778117 18365 raft_consensus.cc:2243] T 2d87ab35f9f343e08090b48099a2e21d P 7ecaaa30057f485f90fc8781084c9ef2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:08:38.778242 18365 raft_consensus.cc:2272] T 2d87ab35f9f343e08090b48099a2e21d P 7ecaaa30057f485f90fc8781084c9ef2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:08:38.778698 18364 tablet_replica.cc:333] T 2d87ab35f9f343e08090b48099a2e21d P bcffd137fdc04dcfbf055e9c4529b211: stopping tablet replica
I20260429 14:08:38.778798 18364 raft_consensus.cc:2243] T 2d87ab35f9f343e08090b48099a2e21d P bcffd137fdc04dcfbf055e9c4529b211 [term 1 LEADER]: Raft consensus shutting down.
I20260429 14:08:38.778903 18364 raft_consensus.cc:2272] T 2d87ab35f9f343e08090b48099a2e21d P bcffd137fdc04dcfbf055e9c4529b211 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:08:38.779320 18364 ts_tablet_manager.cc:1916] T 2d87ab35f9f343e08090b48099a2e21d P bcffd137fdc04dcfbf055e9c4529b211: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:08:38.779767 18365 ts_tablet_manager.cc:1916] T 2d87ab35f9f343e08090b48099a2e21d P 7ecaaa30057f485f90fc8781084c9ef2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:08:38.781387 18365 ts_tablet_manager.cc:1929] T 2d87ab35f9f343e08090b48099a2e21d P 7ecaaa30057f485f90fc8781084c9ef2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 14:08:38.781458 18364 ts_tablet_manager.cc:1929] T 2d87ab35f9f343e08090b48099a2e21d P bcffd137fdc04dcfbf055e9c4529b211: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 14:08:38.781540 18364 log.cc:1199] T 2d87ab35f9f343e08090b48099a2e21d P bcffd137fdc04dcfbf055e9c4529b211: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/wal/wals/2d87ab35f9f343e08090b48099a2e21d
I20260429 14:08:38.781635 18365 log.cc:1199] T 2d87ab35f9f343e08090b48099a2e21d P 7ecaaa30057f485f90fc8781084c9ef2: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/wal/wals/2d87ab35f9f343e08090b48099a2e21d
I20260429 14:08:38.781853 18364 ts_tablet_manager.cc:1950] T 2d87ab35f9f343e08090b48099a2e21d P bcffd137fdc04dcfbf055e9c4529b211: Deleting consensus metadata
I20260429 14:08:38.782078 18365 ts_tablet_manager.cc:1950] T 2d87ab35f9f343e08090b48099a2e21d P 7ecaaa30057f485f90fc8781084c9ef2: Deleting consensus metadata
I20260429 14:08:38.782256 17861 catalog_manager.cc:5002] TS bcffd137fdc04dcfbf055e9c4529b211 (127.2.197.195:36003): tablet 2d87ab35f9f343e08090b48099a2e21d (table test-table [id=6c99f8ea9b2945f7901a6e8eab90456c]) successfully deleted
I20260429 14:08:38.782629 17863 catalog_manager.cc:5002] TS 7ecaaa30057f485f90fc8781084c9ef2 (127.2.197.194:40909): tablet 2d87ab35f9f343e08090b48099a2e21d (table test-table [id=6c99f8ea9b2945f7901a6e8eab90456c]) successfully deleted
W20260429 14:08:38.784790 17861 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.197.193:34871 (error 108)
I20260429 14:08:38.785115 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 18070
I20260429 14:08:38.785707 18072 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:38.785260 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:34871 (local address 127.0.0.1:48674)
0429 14:08:38.785621 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:38.785664 (+ 43us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.193:34871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338}
W20260429 14:08:38.786561 17861 catalog_manager.cc:4729] TS cde63f442d164697b92f79a9c82b955a (127.2.197.193:34871): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2d87ab35f9f343e08090b48099a2e21d: Network error: Client connection negotiation failed: client connection to 127.2.197.193:34871: connect: Connection refused (error 111)
I20260429 14:08:38.791311 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 18206
I20260429 14:08:38.796868 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 17843
I20260429 14:08:38.802963 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:41969
--webserver_interface=127.2.197.254
--webserver_port=35875
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:43751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:41969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:08:38.884553 18367 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:08:38.884786 18367 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:08:38.884814 18367 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:08:38.886587 18367 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:08:38.886662 18367 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:08:38.886678 18367 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:08:38.886691 18367 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:08:38.886703 18367 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:08:38.888588 18367 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:43751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:41969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:41969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=35875
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:08:38.888877 18367 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:08:38.889106 18367 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:08:38.892280 18373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:38.892266 18375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:38.892266 18372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:08:38.892745 18367 server_base.cc:1061] running on GCE node
I20260429 14:08:38.893005 18367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:08:38.893275 18367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:08:38.894493 18367 hybrid_clock.cc:648] HybridClock initialized: now 1777471718894481 us; error 33 us; skew 500 ppm
Apr 29 14:08:38 dist-test-slave-8n77 krb5kdc[17280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471718, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:08:38.897162 18367 init.cc:377] Logged in from keytab as kudu/127.2.197.254@KRBTEST.COM (short username kudu)
I20260429 14:08:38.897729 18367 webserver.cc:492] Webserver started at http://127.2.197.254:35875/ using document root <none> and password file <none>
I20260429 14:08:38.897887 18367 fs_manager.cc:362] Metadata directory not provided
I20260429 14:08:38.897927 18367 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:08:38.899181 18367 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:08:38.899823 18382 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:38.900138 18367 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:08:38.900213 18367 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "af9649ac49224c7e91f0508bd22d69b4"
format_stamp: "Formatted at 2026-04-29 14:08:37 on dist-test-slave-8n77"
server_key: "90e5966293ffe503124c0ca266884ffa"
server_key_iv: "436fbd4071e1926980386e57fdfbf19e"
server_key_version: "encryptionkey@0"
I20260429 14:08:38.900537 18367 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:08:38.911696 18367 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:08:38.913697 18367 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:08:38.913846 18367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:08:38.919695 18367 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:41969
I20260429 14:08:38.919746 18434 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:41969 every 8 connection(s)
I20260429 14:08:38.920136 18367 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:08:38.921329 18435 sys_catalog.cc:263] Verifying existing consensus state
I20260429 14:08:38.921947 18435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4: Bootstrap starting.
I20260429 14:08:38.928124 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 18367
I20260429 14:08:38.928486 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.193:34871
--local_ip_for_outbound_sockets=127.2.197.193
--tserver_master_addrs=127.2.197.254:41969
--webserver_port=43689
--webserver_interface=127.2.197.193
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:43751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260429 14:08:38.932114 18435 log.cc:826] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4: Log is configured to *not* fsync() on all Append() calls
I20260429 14:08:38.934239 18435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260429 14:08:38.934578 18435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4: Bootstrap complete.
I20260429 14:08:38.936592 18435 raft_consensus.cc:359] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9649ac49224c7e91f0508bd22d69b4" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 41969 } }
I20260429 14:08:38.936808 18435 raft_consensus.cc:740] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: af9649ac49224c7e91f0508bd22d69b4, State: Initialized, Role: FOLLOWER
I20260429 14:08:38.936980 18435 consensus_queue.cc:260] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9649ac49224c7e91f0508bd22d69b4" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 41969 } }
I20260429 14:08:38.937054 18435 raft_consensus.cc:399] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:08:38.937103 18435 raft_consensus.cc:493] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:08:38.937148 18435 raft_consensus.cc:3060] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [term 2 FOLLOWER]: Advancing to term 3
I20260429 14:08:38.938001 18435 raft_consensus.cc:515] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9649ac49224c7e91f0508bd22d69b4" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 41969 } }
I20260429 14:08:38.938133 18435 leader_election.cc:304] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af9649ac49224c7e91f0508bd22d69b4; no voters:
I20260429 14:08:38.938359 18435 leader_election.cc:290] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260429 14:08:38.938441 18439 raft_consensus.cc:2804] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [term 3 FOLLOWER]: Leader election won for term 3
I20260429 14:08:38.938606 18439 raft_consensus.cc:697] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [term 3 LEADER]: Becoming Leader. State: Replica: af9649ac49224c7e91f0508bd22d69b4, State: Running, Role: LEADER
I20260429 14:08:38.938613 18435 sys_catalog.cc:565] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:08:38.938797 18439 consensus_queue.cc:237] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9649ac49224c7e91f0508bd22d69b4" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 41969 } }
I20260429 14:08:38.939468 18441 sys_catalog.cc:455] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af9649ac49224c7e91f0508bd22d69b4. Latest consensus state: current_term: 3 leader_uuid: "af9649ac49224c7e91f0508bd22d69b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9649ac49224c7e91f0508bd22d69b4" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 41969 } } }
I20260429 14:08:38.939559 18441 sys_catalog.cc:458] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [sys.catalog]: This master's current role is: LEADER
I20260429 14:08:38.939658 18440 sys_catalog.cc:455] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "af9649ac49224c7e91f0508bd22d69b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9649ac49224c7e91f0508bd22d69b4" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 41969 } } }
I20260429 14:08:38.939715 18440 sys_catalog.cc:458] T 00000000000000000000000000000000 P af9649ac49224c7e91f0508bd22d69b4 [sys.catalog]: This master's current role is: LEADER
I20260429 14:08:38.940009 18449 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:08:38.940716 18449 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:08:38.940856 18449 catalog_manager.cc:1269] Loaded cluster ID: 92e01676e16043d284bee230b2910f0c
I20260429 14:08:38.940905 18449 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
F20260429 14:08:38.941087 18449 catalog_manager.cc:1476] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=RSAPrivateKey error:04093004:rsa routines:old_rsa_priv_decode:RSA lib:../crypto/rsa/rsa_ameth.c:130 error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f105f425dcd google::LogMessage::Fail() at ??:0
@ 0x7f105f429b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f105f4257cc google::LogMessage::Flush() at ??:0
@ 0x7f105f426f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1061efdf06 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f1061f08664 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1060567400 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f106055d3c7 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f10608c76db start_thread at ??:0
@ 0x7f105e97c61f clone at ??:0
W20260429 14:08:39.042905 18437 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:08:39.043176 18437 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:08:39.043215 18437 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:08:39.045763 18437 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:08:39.045845 18437 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:08:39.045898 18437 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.193
I20260429 14:08:39.048020 18437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:43751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.193:34871
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.197.193
--webserver_port=43689
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:41969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.193
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:08:39.048277 18437 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:08:39.048497 18437 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:08:39.051416 18474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:39.051563 18471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:08:39.051380 18437 server_base.cc:1061] running on GCE node
W20260429 14:08:39.051378 18472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:08:39.051856 18437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:08:39.052075 18437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:08:39.053246 18437 hybrid_clock.cc:648] HybridClock initialized: now 1777471719053218 us; error 44 us; skew 500 ppm
Apr 29 14:08:39 dist-test-slave-8n77 krb5kdc[17280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471719, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:08:39.055749 18437 init.cc:377] Logged in from keytab as kudu/127.2.197.193@KRBTEST.COM (short username kudu)
I20260429 14:08:39.056320 18437 webserver.cc:492] Webserver started at http://127.2.197.193:43689/ using document root <none> and password file <none>
I20260429 14:08:39.056531 18437 fs_manager.cc:362] Metadata directory not provided
I20260429 14:08:39.056592 18437 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:08:39.057824 18437 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:08:39.058487 18481 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:39.058697 18437 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:08:39.058769 18437 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/wal
uuid: "cde63f442d164697b92f79a9c82b955a"
format_stamp: "Formatted at 2026-04-29 14:08:37 on dist-test-slave-8n77"
server_key: "38689aa1c0fb4b189491af6bab1b4580"
server_key_iv: "ff49f67e55bef8208152272e9b77cdcd"
server_key_version: "encryptionkey@0"
I20260429 14:08:39.059065 18437 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:08:39.067589 18437 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:08:39.069656 18437 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:08:39.069813 18437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:08:39.070036 18437 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:08:39.070581 18488 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260429 14:08:39.071782 18437 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260429 14:08:39.071854 18437 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260429 14:08:39.071915 18437 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260429 14:08:39.072482 18437 ts_tablet_manager.cc:616] Registered 1 tablets
I20260429 14:08:39.072542 18437 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260429 14:08:39.072630 18488 tablet_bootstrap.cc:492] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a: Bootstrap starting.
I20260429 14:08:39.081537 18437 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.193:34871
I20260429 14:08:39.081570 18595 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.193:34871 every 8 connection(s)
I20260429 14:08:39.081991 18437 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
I20260429 14:08:39.084246 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 18437
I20260429 14:08:39.084546 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.194:40909
--local_ip_for_outbound_sockets=127.2.197.194
--tserver_master_addrs=127.2.197.254:41969
--webserver_port=39345
--webserver_interface=127.2.197.194
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:43751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260429 14:08:39.084589 18488 log.cc:826] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a: Log is configured to *not* fsync() on all Append() calls
I20260429 14:08:39.086658 18488 tablet_bootstrap.cc:492] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260429 14:08:39.086993 18488 tablet_bootstrap.cc:492] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a: Bootstrap complete.
I20260429 14:08:39.088038 18488 ts_tablet_manager.cc:1403] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.009s
I20260429 14:08:39.089406 18488 raft_consensus.cc:359] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcffd137fdc04dcfbf055e9c4529b211" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36003 } } peers { permanent_uuid: "7ecaaa30057f485f90fc8781084c9ef2" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40909 } } peers { permanent_uuid: "cde63f442d164697b92f79a9c82b955a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34871 } }
I20260429 14:08:39.089743 18488 raft_consensus.cc:740] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cde63f442d164697b92f79a9c82b955a, State: Initialized, Role: FOLLOWER
I20260429 14:08:39.089876 18488 consensus_queue.cc:260] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcffd137fdc04dcfbf055e9c4529b211" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36003 } } peers { permanent_uuid: "7ecaaa30057f485f90fc8781084c9ef2" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40909 } } peers { permanent_uuid: "cde63f442d164697b92f79a9c82b955a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34871 } }
I20260429 14:08:39.090138 18488 ts_tablet_manager.cc:1434] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260429 14:08:39.167773 18600 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:08:39.168051 18600 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:08:39.168103 18600 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:08:39.170039 18600 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:08:39.170121 18600 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:08:39.170169 18600 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.194
I20260429 14:08:39.172153 18600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:43751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.194:40909
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.197.194
--webserver_port=39345
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:41969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.194
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:08:39.172449 18600 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:08:39.172832 18600 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:08:39.175566 18607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:39.175711 18609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:39.175580 18606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:08:39.175961 18600 server_base.cc:1061] running on GCE node
I20260429 14:08:39.176169 18600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:08:39.176429 18600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:08:39.177610 18600 hybrid_clock.cc:648] HybridClock initialized: now 1777471719177581 us; error 38 us; skew 500 ppm
Apr 29 14:08:39 dist-test-slave-8n77 krb5kdc[17280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471719, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:08:39.180771 18600 init.cc:377] Logged in from keytab as kudu/127.2.197.194@KRBTEST.COM (short username kudu)
I20260429 14:08:39.181403 18600 webserver.cc:492] Webserver started at http://127.2.197.194:39345/ using document root <none> and password file <none>
I20260429 14:08:39.181619 18600 fs_manager.cc:362] Metadata directory not provided
I20260429 14:08:39.181685 18600 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:08:39.183323 18600 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:08:39.183972 18616 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:39.184182 18600 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260429 14:08:39.184265 18600 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/wal
uuid: "7ecaaa30057f485f90fc8781084c9ef2"
format_stamp: "Formatted at 2026-04-29 14:08:37 on dist-test-slave-8n77"
server_key: "1873e40a0ca76054160f00f6125fd435"
server_key_iv: "60ad4af3288d8fb69c32ab12bab2d918"
server_key_version: "encryptionkey@0"
I20260429 14:08:39.184634 18600 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260429 14:08:39.186923 18598 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:08:39.082914 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.193:37727)
0429 14:08:39.083115 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:39.083151 (+ 36us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:39.083618 (+ 467us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:39.186778 (+103160us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":123,"threads_started":1}
W20260429 14:08:39.187211 18596 heartbeater.cc:646] Failed to heartbeat to 127.2.197.254:41969 (0 consecutive failures): Network error: Failed to ping master at 127.2.197.254:41969: Client connection negotiation failed: client connection to 127.2.197.254:41969: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260429 14:08:39.187640 18598 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:39.187498 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.193:59335)
0429 14:08:39.187560 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:39.187613 (+ 53us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260429 14:08:39.187987 18598 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:39.187881 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.193:47545)
0429 14:08:39.187936 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:39.187967 (+ 31us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260429 14:08:39.188153 18596 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260429 14:08:39.207065 18600 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:08:39.209370 18600 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:08:39.209488 18600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:08:39.209686 18600 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:08:39.210014 18600 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:08:39.210049 18600 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:39.210072 18600 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:08:39.210121 18600 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:39.216436 18600 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.194:40909
I20260429 14:08:39.216491 18729 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.194:40909 every 8 connection(s)
I20260429 14:08:39.216820 18600 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
I20260429 14:08:39.218116 18732 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:39.217667 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.194:56915)
0429 14:08:39.217839 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:39.217914 (+ 75us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":74,"threads_started":1}
W20260429 14:08:39.218508 18730 heartbeater.cc:646] Failed to heartbeat to 127.2.197.254:41969 (0 consecutive failures): Network error: Failed to ping master at 127.2.197.254:41969: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
I20260429 14:08:39.218890 18732 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:39.218734 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.194:44039)
0429 14:08:39.218830 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:39.218865 (+ 35us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260429 14:08:39.219442 18732 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:39.219209 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.194:58443)
0429 14:08:39.219385 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:39.219420 (+ 35us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
W20260429 14:08:39.219663 18730 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260429 14:08:39.222993 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 18600
I20260429 14:08:39.223299 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.195:36003
--local_ip_for_outbound_sockets=127.2.197.195
--tserver_master_addrs=127.2.197.254:41969
--webserver_port=38219
--webserver_interface=127.2.197.195
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:43751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:08:39.304184 18733 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:08:39.304457 18733 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:08:39.304498 18733 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:08:39.306102 18733 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:08:39.306208 18733 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:08:39.306255 18733 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.195
I20260429 14:08:39.308185 18733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:43751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.195:36003
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.197.195
--webserver_port=38219
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:41969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.195
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:08:39.308480 18733 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:08:39.308681 18733 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:08:39.311292 18738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:08:39.311359 18733 server_base.cc:1061] running on GCE node
W20260429 14:08:39.311300 18741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:39.311429 18739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:08:39.311790 18733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:08:39.311990 18733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:08:39.313156 18733 hybrid_clock.cc:648] HybridClock initialized: now 1777471719313144 us; error 31 us; skew 500 ppm
Apr 29 14:08:39 dist-test-slave-8n77 krb5kdc[17280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471719, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:08:39.316032 18733 init.cc:377] Logged in from keytab as kudu/127.2.197.195@KRBTEST.COM (short username kudu)
I20260429 14:08:39.316651 18733 webserver.cc:492] Webserver started at http://127.2.197.195:38219/ using document root <none> and password file <none>
I20260429 14:08:39.316862 18733 fs_manager.cc:362] Metadata directory not provided
I20260429 14:08:39.316927 18733 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:08:39.318354 18733 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:08:39.319072 18748 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:39.319303 18733 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:08:39.319398 18733 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/wal
uuid: "bcffd137fdc04dcfbf055e9c4529b211"
format_stamp: "Formatted at 2026-04-29 14:08:37 on dist-test-slave-8n77"
server_key: "25e53cf0c3badb473a63e257bf278ae6"
server_key_iv: "6c0059b5d7a1c3086c5fb24f1b46f208"
server_key_version: "encryptionkey@0"
I20260429 14:08:39.319712 18733 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:08:39.343312 18733 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:08:39.345417 18733 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:08:39.345628 18733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:08:39.345891 18733 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:08:39.346215 18733 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:08:39.346272 18733 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:39.346364 18733 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:08:39.346388 18733 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:39.352779 18733 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.195:36003
I20260429 14:08:39.352818 18861 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.195:36003 every 8 connection(s)
I20260429 14:08:39.353133 18733 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
I20260429 14:08:39.354085 18864 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:39.353781 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.195:39249)
0429 14:08:39.353912 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:39.353989 (+ 77us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":57,"threads_started":1}
W20260429 14:08:39.354351 18862 heartbeater.cc:646] Failed to heartbeat to 127.2.197.254:41969 (0 consecutive failures): Network error: Failed to ping master at 127.2.197.254:41969: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
I20260429 14:08:39.354689 18864 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:39.354540 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.195:48809)
0429 14:08:39.354644 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:39.354672 (+ 28us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260429 14:08:39.355086 18864 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:39.354975 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.195:53075)
0429 14:08:39.355036 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:39.355065 (+ 29us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260429 14:08:39.355263 18862 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260429 14:08:39.358585 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 18733
I20260429 14:08:40.188908 18866 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:40.188597 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.193:48833)
0429 14:08:40.188810 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:40.188860 (+ 50us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":85,"threads_started":1}
I20260429 14:08:40.220340 18867 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:40.220081 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.194:42379)
0429 14:08:40.220235 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:40.220278 (+ 43us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":74,"threads_started":1}
I20260429 14:08:40.356081 18868 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:40.355743 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.195:35227)
0429 14:08:40.355888 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:40.356007 (+ 119us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":102,"threads_started":1}
I20260429 14:08:40.740453 18869 raft_consensus.cc:493] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:08:40.740598 18869 raft_consensus.cc:515] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcffd137fdc04dcfbf055e9c4529b211" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36003 } } peers { permanent_uuid: "7ecaaa30057f485f90fc8781084c9ef2" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40909 } } peers { permanent_uuid: "cde63f442d164697b92f79a9c82b955a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34871 } }
I20260429 14:08:40.740871 18869 leader_election.cc:290] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcffd137fdc04dcfbf055e9c4529b211 (127.2.197.195:36003), 7ecaaa30057f485f90fc8781084c9ef2 (127.2.197.194:40909)
Apr 29 14:08:40 dist-test-slave-8n77 krb5kdc[17280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471719, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for kudu/127.2.197.194@KRBTEST.COM
Apr 29 14:08:40 dist-test-slave-8n77 krb5kdc[17280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471719, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for kudu/127.2.197.195@KRBTEST.COM
I20260429 14:08:40.757712 18872 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:40.741199 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:38543 (local address 127.2.197.195:36003)
0429 14:08:40.741364 (+ 165us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:40.741385 (+ 21us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:40.741398 (+ 13us) server_negotiation.cc:408] Connection header received
0429 14:08:40.741949 (+ 551us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:40.741953 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:40.742006 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:40.742243 (+ 237us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:40.743737 (+ 1494us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:40.745401 (+ 1664us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:40.746393 (+ 992us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:40.746612 (+ 219us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:40.753639 (+ 7027us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:40.753664 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:40.753664 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:40.753683 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:40.756030 (+ 2347us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:40.756452 (+ 422us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:40.756455 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:40.756456 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:40.756607 (+ 151us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:40.756809 (+ 202us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:40.756809 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:40.756810 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:40.757025 (+ 215us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:40.757083 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:40.757472 (+ 389us) server_negotiation.cc:300] Negotiation successful
0429 14:08:40.757604 (+ 132us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":72,"threads_started":1}
I20260429 14:08:40.757730 18873 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:40.741233 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:37697 (local address 127.2.197.194:40909)
0429 14:08:40.741364 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:40.741373 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:40.741395 (+ 22us) server_negotiation.cc:408] Connection header received
0429 14:08:40.741963 (+ 568us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:40.741965 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:40.742089 (+ 124us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:40.742243 (+ 154us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:40.743737 (+ 1494us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:40.745401 (+ 1664us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:40.746719 (+ 1318us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:40.746903 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:40.753207 (+ 6304us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:40.753243 (+ 36us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:40.753244 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:40.753262 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:40.755714 (+ 2452us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:40.756183 (+ 469us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:40.756185 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:40.756186 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:40.756235 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:40.756562 (+ 327us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:40.756563 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:40.756564 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:40.756865 (+ 301us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:40.756949 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:40.757464 (+ 515us) server_negotiation.cc:300] Negotiation successful
0429 14:08:40.757602 (+ 138us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":49,"threads_started":1}
I20260429 14:08:40.757797 18871 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:40.741083 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:36003 (local address 127.2.197.193:38543)
0429 14:08:40.741200 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:40.741214 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:40.741274 (+ 60us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:40.742337 (+ 1063us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:40.742340 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:40.742991 (+ 651us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:08:40.743444 (+ 453us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:40.743449 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:40.745623 (+ 2174us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:40.745625 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:40.746234 (+ 609us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:40.746236 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:40.746401 (+ 165us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:40.751087 (+ 4686us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:08:40.751101 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:08:40.753508 (+ 2407us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:08:40.756126 (+ 2618us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:40.756127 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:40.756128 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:40.756303 (+ 175us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:40.756707 (+ 404us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:40.756708 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:40.756708 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:40.756738 (+ 30us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:40.757106 (+ 368us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:08:40.757107 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:08:40.757141 (+ 34us) client_negotiation.cc:770] Sending connection context
0429 14:08:40.757248 (+ 107us) client_negotiation.cc:241] Negotiation successful
0429 14:08:40.757557 (+ 309us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":14,"thread_start_us":40,"threads_started":1}
I20260429 14:08:40.758152 18870 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:40.741083 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:40909 (local address 127.2.197.193:37697)
0429 14:08:40.741200 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:40.741214 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:40.741274 (+ 60us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:40.742337 (+ 1063us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:40.742340 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:40.742987 (+ 647us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:08:40.743444 (+ 457us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:40.743449 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:40.745640 (+ 2191us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:40.745641 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:40.746234 (+ 593us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:40.746236 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:40.746408 (+ 172us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:40.751128 (+ 4720us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:08:40.751134 (+ 6us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:08:40.753066 (+ 1932us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:08:40.755826 (+ 2760us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:40.755833 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:40.755835 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:40.756048 (+ 213us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:40.756304 (+ 256us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:40.756305 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:40.756305 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:40.756435 (+ 130us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:40.756960 (+ 525us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:08:40.756965 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:08:40.757116 (+ 151us) client_negotiation.cc:770] Sending connection context
0429 14:08:40.757331 (+ 215us) client_negotiation.cc:241] Negotiation successful
0429 14:08:40.757635 (+ 304us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":127,"thread_start_us":51,"threads_started":1}
W20260429 14:08:40.759091 18483 leader_election.cc:343] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bcffd137fdc04dcfbf055e9c4529b211 (127.2.197.195:36003): Not found: Tablet not found: 2d87ab35f9f343e08090b48099a2e21d
W20260429 14:08:40.760042 18485 leader_election.cc:343] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ecaaa30057f485f90fc8781084c9ef2 (127.2.197.194:40909): Not found: Tablet not found: 2d87ab35f9f343e08090b48099a2e21d
I20260429 14:08:40.760161 18485 leader_election.cc:304] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cde63f442d164697b92f79a9c82b955a; no voters: 7ecaaa30057f485f90fc8781084c9ef2, bcffd137fdc04dcfbf055e9c4529b211
I20260429 14:08:40.760360 18869 raft_consensus.cc:2749] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260429 14:08:41.189700 18870 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:41.189516 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.193:34361)
0429 14:08:41.189596 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:41.189659 (+ 63us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260429 14:08:41.221262 18874 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:41.220954 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.194:38769)
0429 14:08:41.221150 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:41.221216 (+ 66us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":72,"threads_started":1}
I20260429 14:08:41.356973 18875 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:41.356755 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.195:43893)
0429 14:08:41.356887 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:41.356928 (+ 41us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":62,"threads_started":1}
I20260429 14:08:42.190622 18876 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:42.190332 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.193:58237)
0429 14:08:42.190519 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:42.190563 (+ 44us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":106,"threads_started":1}
I20260429 14:08:42.222215 18877 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:42.221919 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.194:54499)
0429 14:08:42.222131 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:42.222168 (+ 37us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":152,"threads_started":1}
I20260429 14:08:42.357838 18878 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:42.357580 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.195:53591)
0429 14:08:42.357742 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:42.357790 (+ 48us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":81,"threads_started":1}
I20260429 14:08:42.520391 18879 raft_consensus.cc:493] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:08:42.520591 18879 raft_consensus.cc:515] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcffd137fdc04dcfbf055e9c4529b211" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36003 } } peers { permanent_uuid: "7ecaaa30057f485f90fc8781084c9ef2" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40909 } } peers { permanent_uuid: "cde63f442d164697b92f79a9c82b955a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34871 } }
I20260429 14:08:42.520823 18879 leader_election.cc:290] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcffd137fdc04dcfbf055e9c4529b211 (127.2.197.195:36003), 7ecaaa30057f485f90fc8781084c9ef2 (127.2.197.194:40909)
W20260429 14:08:42.521229 18483 leader_election.cc:343] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bcffd137fdc04dcfbf055e9c4529b211 (127.2.197.195:36003): Not found: Tablet not found: 2d87ab35f9f343e08090b48099a2e21d
W20260429 14:08:42.521291 18485 leader_election.cc:343] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ecaaa30057f485f90fc8781084c9ef2 (127.2.197.194:40909): Not found: Tablet not found: 2d87ab35f9f343e08090b48099a2e21d
I20260429 14:08:42.521332 18485 leader_election.cc:304] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cde63f442d164697b92f79a9c82b955a; no voters: 7ecaaa30057f485f90fc8781084c9ef2, bcffd137fdc04dcfbf055e9c4529b211
I20260429 14:08:42.521458 18879 raft_consensus.cc:2749] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260429 14:08:43.191612 18880 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:43.191266 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.193:57193)
0429 14:08:43.191514 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:43.191564 (+ 50us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":108,"threads_started":1}
I20260429 14:08:43.223217 18881 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:43.222953 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.194:49541)
0429 14:08:43.223115 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:43.223163 (+ 48us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":111,"threads_started":1}
I20260429 14:08:43.358794 18882 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:43.358564 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.195:59603)
0429 14:08:43.358711 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:43.358750 (+ 39us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":85,"threads_started":1}
I20260429 14:08:44.192489 18883 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:44.192207 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.193:34559)
0429 14:08:44.192364 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:44.192441 (+ 77us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":67,"threads_started":1}
I20260429 14:08:44.224269 18884 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:44.224053 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.194:45023)
0429 14:08:44.224178 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:44.224223 (+ 45us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260429 14:08:44.359716 18885 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:44.359413 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.195:38653)
0429 14:08:44.359571 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:44.359656 (+ 85us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":71,"threads_started":1}
I20260429 14:08:45.193637 18886 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:45.193179 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.193:59515)
0429 14:08:45.193461 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:45.193560 (+ 99us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":199,"threads_started":1}
I20260429 14:08:45.225205 18887 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:45.224961 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.194:45429)
0429 14:08:45.225091 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:45.225135 (+ 44us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260429 14:08:45.360479 18888 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:45.360207 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.195:37587)
0429 14:08:45.360387 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:45.360431 (+ 44us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":65,"threads_started":1}
I20260429 14:08:45.494545 18889 raft_consensus.cc:493] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:08:45.494645 18889 raft_consensus.cc:515] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcffd137fdc04dcfbf055e9c4529b211" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36003 } } peers { permanent_uuid: "7ecaaa30057f485f90fc8781084c9ef2" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40909 } } peers { permanent_uuid: "cde63f442d164697b92f79a9c82b955a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34871 } }
I20260429 14:08:45.494820 18889 leader_election.cc:290] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcffd137fdc04dcfbf055e9c4529b211 (127.2.197.195:36003), 7ecaaa30057f485f90fc8781084c9ef2 (127.2.197.194:40909)
W20260429 14:08:45.495250 18483 leader_election.cc:343] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bcffd137fdc04dcfbf055e9c4529b211 (127.2.197.195:36003): Not found: Tablet not found: 2d87ab35f9f343e08090b48099a2e21d
W20260429 14:08:45.495383 18485 leader_election.cc:343] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ecaaa30057f485f90fc8781084c9ef2 (127.2.197.194:40909): Not found: Tablet not found: 2d87ab35f9f343e08090b48099a2e21d
I20260429 14:08:45.495460 18485 leader_election.cc:304] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cde63f442d164697b92f79a9c82b955a; no voters: 7ecaaa30057f485f90fc8781084c9ef2, bcffd137fdc04dcfbf055e9c4529b211
I20260429 14:08:45.495657 18889 raft_consensus.cc:2749] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260429 14:08:46.199281 18890 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:46.199003 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.193:48801)
0429 14:08:46.199154 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:46.199202 (+ 48us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":75,"threads_started":1}
I20260429 14:08:46.226089 18891 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:46.225817 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.194:41579)
0429 14:08:46.226000 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:46.226043 (+ 43us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":81,"threads_started":1}
I20260429 14:08:46.361410 18892 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:46.361133 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.195:41719)
0429 14:08:46.361306 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:46.361358 (+ 52us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":67,"threads_started":1}
I20260429 14:08:47.200140 18893 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:47.199885 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.193:33819)
0429 14:08:47.200060 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:47.200091 (+ 31us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":96,"threads_started":1}
I20260429 14:08:47.226797 18894 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:47.226566 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.194:50677)
0429 14:08:47.226717 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:47.226752 (+ 35us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":103,"threads_started":1}
I20260429 14:08:47.362217 18895 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:47.361989 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.195:44795)
0429 14:08:47.362131 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:47.362171 (+ 40us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":60,"threads_started":1}
I20260429 14:08:48.200999 18896 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:48.200734 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.193:43865)
0429 14:08:48.200903 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:48.200950 (+ 47us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":66,"threads_started":1}
I20260429 14:08:48.227536 18897 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:48.227320 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.194:35167)
0429 14:08:48.227456 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:48.227491 (+ 35us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
I20260429 14:08:48.363232 18898 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:48.362981 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.195:41919)
0429 14:08:48.363137 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:48.363185 (+ 48us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":74,"threads_started":1}
I20260429 14:08:48.724114 18899 raft_consensus.cc:493] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:08:48.724234 18899 raft_consensus.cc:515] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcffd137fdc04dcfbf055e9c4529b211" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36003 } } peers { permanent_uuid: "7ecaaa30057f485f90fc8781084c9ef2" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40909 } } peers { permanent_uuid: "cde63f442d164697b92f79a9c82b955a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34871 } }
I20260429 14:08:48.724453 18899 leader_election.cc:290] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcffd137fdc04dcfbf055e9c4529b211 (127.2.197.195:36003), 7ecaaa30057f485f90fc8781084c9ef2 (127.2.197.194:40909)
W20260429 14:08:48.725011 18483 leader_election.cc:343] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bcffd137fdc04dcfbf055e9c4529b211 (127.2.197.195:36003): Not found: Tablet not found: 2d87ab35f9f343e08090b48099a2e21d
W20260429 14:08:48.725081 18485 leader_election.cc:343] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ecaaa30057f485f90fc8781084c9ef2 (127.2.197.194:40909): Not found: Tablet not found: 2d87ab35f9f343e08090b48099a2e21d
I20260429 14:08:48.725116 18485 leader_election.cc:304] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cde63f442d164697b92f79a9c82b955a; no voters: 7ecaaa30057f485f90fc8781084c9ef2, bcffd137fdc04dcfbf055e9c4529b211
I20260429 14:08:48.725194 18899 raft_consensus.cc:2749] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260429 14:08:49.201862 18900 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:49.201609 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.193:55303)
0429 14:08:49.201770 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:49.201811 (+ 41us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":70,"threads_started":1}
I20260429 14:08:49.228271 18901 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:49.228057 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.194:34203)
0429 14:08:49.228181 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:49.228223 (+ 42us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
I20260429 14:08:49.364082 18902 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:49.363845 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.195:34815)
0429 14:08:49.363989 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:49.364034 (+ 45us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":66,"threads_started":1}
I20260429 14:08:50.202793 18903 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:50.202562 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.193:48895)
0429 14:08:50.202699 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:50.202745 (+ 46us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":65,"threads_started":1}
I20260429 14:08:50.229338 18904 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:50.228985 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.194:60703)
0429 14:08:50.229200 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:50.229252 (+ 52us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":90,"threads_started":1}
I20260429 14:08:50.364917 18905 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:50.364669 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.195:41089)
0429 14:08:50.364821 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:50.364866 (+ 45us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":61,"threads_started":1}
I20260429 14:08:51.203703 18906 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:51.203436 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.193:33527)
0429 14:08:51.203594 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:51.203655 (+ 61us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":74,"threads_started":1}
I20260429 14:08:51.230258 18907 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:51.230006 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.194:34507)
0429 14:08:51.230172 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:51.230211 (+ 39us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":66,"threads_started":1}
I20260429 14:08:51.365830 18908 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:51.365550 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.195:57865)
0429 14:08:51.365716 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:51.365762 (+ 46us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":79,"threads_started":1}
I20260429 14:08:51.383420 18909 raft_consensus.cc:493] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:08:51.383525 18909 raft_consensus.cc:515] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcffd137fdc04dcfbf055e9c4529b211" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36003 } } peers { permanent_uuid: "7ecaaa30057f485f90fc8781084c9ef2" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 40909 } } peers { permanent_uuid: "cde63f442d164697b92f79a9c82b955a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34871 } }
I20260429 14:08:51.383865 18909 leader_election.cc:290] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcffd137fdc04dcfbf055e9c4529b211 (127.2.197.195:36003), 7ecaaa30057f485f90fc8781084c9ef2 (127.2.197.194:40909)
W20260429 14:08:51.384301 18483 leader_election.cc:343] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bcffd137fdc04dcfbf055e9c4529b211 (127.2.197.195:36003): Not found: Tablet not found: 2d87ab35f9f343e08090b48099a2e21d
W20260429 14:08:51.384382 18485 leader_election.cc:343] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ecaaa30057f485f90fc8781084c9ef2 (127.2.197.194:40909): Not found: Tablet not found: 2d87ab35f9f343e08090b48099a2e21d
I20260429 14:08:51.384426 18485 leader_election.cc:304] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cde63f442d164697b92f79a9c82b955a; no voters: 7ecaaa30057f485f90fc8781084c9ef2, bcffd137fdc04dcfbf055e9c4529b211
I20260429 14:08:51.384526 18909 raft_consensus.cc:2749] T 2d87ab35f9f343e08090b48099a2e21d P cde63f442d164697b92f79a9c82b955a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260429 14:08:52.204631 18910 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:52.204347 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.193:52683)
0429 14:08:52.204529 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:52.204573 (+ 44us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":73,"threads_started":1}
I20260429 14:08:52.231148 18911 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:52.230903 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.194:48315)
0429 14:08:52.231055 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:52.231100 (+ 45us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":68,"threads_started":1}
I20260429 14:08:52.366710 18912 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:52.366466 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.195:52117)
0429 14:08:52.366618 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:52.366664 (+ 46us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":109,"threads_started":1}
I20260429 14:08:53.205654 18913 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:53.205357 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.193:59697)
0429 14:08:53.205558 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:53.205604 (+ 46us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":98,"threads_started":1}
I20260429 14:08:53.232230 18914 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:53.231856 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.194:59701)
0429 14:08:53.232082 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:53.232155 (+ 73us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":80,"threads_started":1}
I20260429 14:08:53.367484 18915 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:53.367258 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.195:50383)
0429 14:08:53.367397 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:53.367436 (+ 39us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
I20260429 14:08:54.206842 18916 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:54.206552 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.193:43383)
0429 14:08:54.206750 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:54.206794 (+ 44us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":98,"threads_started":1}
I20260429 14:08:54.233290 18917 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:54.233031 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.194:54935)
0429 14:08:54.233177 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:54.233224 (+ 47us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
I20260429 14:08:54.362684 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 18437
I20260429 14:08:54.368014 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 18600
I20260429 14:08:54.368857 18918 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:54.368056 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:41969 (local address 127.2.197.195:51515)
0429 14:08:54.368743 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:54.368782 (+ 39us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":59,"threads_started":1}
I20260429 14:08:54.373144 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 18733
2026-04-29T14:08:54Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19570 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:08:54 dist-test-slave-8n77 krb5kdc[18921](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:08:54 dist-test-slave-8n77 krb5kdc[18921](info): set up 2 sockets
Apr 29 14:08:54 dist-test-slave-8n77 krb5kdc[18921](info): commencing operation
krb5kdc: starting...
W20260429 14:08:56.436280 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.042s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:08:56 dist-test-slave-8n77 krb5kdc[18921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:08:56Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:08:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:08:56.585863 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:46655
--webserver_interface=127.2.197.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:39857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:46655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:08:56.664722 18937 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:08:56.664973 18937 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:08:56.665009 18937 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:08:56.666605 18937 flags.cc:432] Enabled experimental flag: --authn_token_validity_seconds=20
W20260429 14:08:56.666687 18937 flags.cc:432] Enabled experimental flag: --authz_token_validity_seconds=20
W20260429 14:08:56.666708 18937 flags.cc:432] Enabled experimental flag: --tsk_rotation_seconds=20
W20260429 14:08:56.666728 18937 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:08:56.666757 18937 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:08:56.666788 18937 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:08:56.666813 18937 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:08:56.666846 18937 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:08:56.668505 18937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:39857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:46655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:46655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:08:56.668793 18937 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:08:56.668987 18937 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:08:56.671653 18943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:56.671726 18942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:56.671659 18945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:08:56.671831 18937 server_base.cc:1061] running on GCE node
I20260429 14:08:56.672366 18937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:08:56.672786 18937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:08:56.673998 18937 hybrid_clock.cc:648] HybridClock initialized: now 1777471736673979 us; error 39 us; skew 500 ppm
Apr 29 14:08:56 dist-test-slave-8n77 krb5kdc[18921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471736, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:08:56.676680 18937 init.cc:377] Logged in from keytab as kudu/127.2.197.254@KRBTEST.COM (short username kudu)
I20260429 14:08:56.677294 18937 webserver.cc:492] Webserver started at http://127.2.197.254:32977/ using document root <none> and password file <none>
I20260429 14:08:56.677507 18937 fs_manager.cc:362] Metadata directory not provided
I20260429 14:08:56.677548 18937 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:08:56.677660 18937 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:08:56.678704 18937 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "a17d1d014a204c9b99cb95ef72965f22"
format_stamp: "Formatted at 2026-04-29 14:08:56 on dist-test-slave-8n77"
server_key: "8d0393de5cf257c663414fd2a24b5bda"
server_key_iv: "6ab51389eb13d6c82a55acf23ecdad1a"
server_key_version: "encryptionkey@0"
I20260429 14:08:56.679032 18937 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "a17d1d014a204c9b99cb95ef72965f22"
format_stamp: "Formatted at 2026-04-29 14:08:56 on dist-test-slave-8n77"
server_key: "8d0393de5cf257c663414fd2a24b5bda"
server_key_iv: "6ab51389eb13d6c82a55acf23ecdad1a"
server_key_version: "encryptionkey@0"
I20260429 14:08:56.680337 18937 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:08:56.681141 18952 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:56.681387 18937 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:08:56.681483 18937 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "a17d1d014a204c9b99cb95ef72965f22"
format_stamp: "Formatted at 2026-04-29 14:08:56 on dist-test-slave-8n77"
server_key: "8d0393de5cf257c663414fd2a24b5bda"
server_key_iv: "6ab51389eb13d6c82a55acf23ecdad1a"
server_key_version: "encryptionkey@0"
I20260429 14:08:56.681565 18937 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:08:56.710898 18937 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:08:56.713038 18937 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:08:56.713214 18937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:08:56.717311 18937 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:46655
I20260429 14:08:56.717471 19004 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:46655 every 8 connection(s)
I20260429 14:08:56.717730 18937 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:08:56.718357 19005 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:08:56.720752 19005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22: Bootstrap starting.
I20260429 14:08:56.721036 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 18937
I20260429 14:08:56.721163 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:08:56.721478 2839 external_mini_cluster.cc:1468] Setting key a729b9f476d87dec496b65f8886171f0
I20260429 14:08:56.721613 19005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22: Neither blocks nor log segments found. Creating new log.
I20260429 14:08:56.722352 19005 log.cc:826] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22: Log is configured to *not* fsync() on all Append() calls
I20260429 14:08:56.723129 19005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22: No bootstrap required, opened a new log
I20260429 14:08:56.724543 19005 raft_consensus.cc:359] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a17d1d014a204c9b99cb95ef72965f22" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 46655 } }
I20260429 14:08:56.724692 19005 raft_consensus.cc:385] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:08:56.724725 19005 raft_consensus.cc:740] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a17d1d014a204c9b99cb95ef72965f22, State: Initialized, Role: FOLLOWER
I20260429 14:08:56.724822 19005 consensus_queue.cc:260] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a17d1d014a204c9b99cb95ef72965f22" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 46655 } }
I20260429 14:08:56.724903 19005 raft_consensus.cc:399] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:08:56.724938 19005 raft_consensus.cc:493] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:08:56.725016 19005 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:08:56.725672 19005 raft_consensus.cc:515] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a17d1d014a204c9b99cb95ef72965f22" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 46655 } }
I20260429 14:08:56.725785 19005 leader_election.cc:304] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a17d1d014a204c9b99cb95ef72965f22; no voters:
I20260429 14:08:56.725986 19005 leader_election.cc:290] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:08:56.726066 19010 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:08:56.726212 19010 raft_consensus.cc:697] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [term 1 LEADER]: Becoming Leader. State: Replica: a17d1d014a204c9b99cb95ef72965f22, State: Running, Role: LEADER
I20260429 14:08:56.726253 19005 sys_catalog.cc:565] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:08:56.726347 19010 consensus_queue.cc:237] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a17d1d014a204c9b99cb95ef72965f22" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 46655 } }
I20260429 14:08:56.726758 19011 sys_catalog.cc:455] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a17d1d014a204c9b99cb95ef72965f22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a17d1d014a204c9b99cb95ef72965f22" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 46655 } } }
I20260429 14:08:56.726837 19011 sys_catalog.cc:458] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [sys.catalog]: This master's current role is: LEADER
I20260429 14:08:56.727128 19012 sys_catalog.cc:455] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a17d1d014a204c9b99cb95ef72965f22. Latest consensus state: current_term: 1 leader_uuid: "a17d1d014a204c9b99cb95ef72965f22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a17d1d014a204c9b99cb95ef72965f22" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 46655 } } }
I20260429 14:08:56.727211 19012 sys_catalog.cc:458] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [sys.catalog]: This master's current role is: LEADER
I20260429 14:08:56.727623 19015 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
Apr 29 14:08:56 dist-test-slave-8n77 krb5kdc[18921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:08:56.728098 19015 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:08:56.732235 19008 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:56.722022 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:59240 (local address 127.2.197.254:46655)
0429 14:08:56.722196 (+ 174us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:56.722208 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:56.722228 (+ 20us) server_negotiation.cc:408] Connection header received
0429 14:08:56.722661 (+ 433us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:56.722668 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:56.722708 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:56.722815 (+ 107us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:56.723645 (+ 830us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:56.724807 (+ 1162us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:56.725540 (+ 733us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:56.725802 (+ 262us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:56.728557 (+ 2755us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:56.728565 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:56.728565 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:56.728577 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:56.730783 (+ 2206us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:56.731176 (+ 393us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:56.731179 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:56.731179 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:56.731218 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:56.731413 (+ 195us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:56.731414 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:56.731415 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:56.731614 (+ 199us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:56.731753 (+ 139us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:56.731896 (+ 143us) server_negotiation.cc:300] Negotiation successful
0429 14:08:56.732003 (+ 107us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":95,"threads_started":1}
I20260429 14:08:56.732703 19015 catalog_manager.cc:1357] Generated new cluster ID: 97b25306dfc84bfcbe5e5f50aca04bc0
I20260429 14:08:56.732766 19015 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:08:56.747031 19015 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:08:56.747610 19015 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:08:56.752524 19015 catalog_manager.cc:6044] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22: Generated new TSK 0
I20260429 14:08:56.752737 19015 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:08:56.810132 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.193:0
--local_ip_for_outbound_sockets=127.2.197.193
--webserver_interface=127.2.197.193
--webserver_port=0
--tserver_master_addrs=127.2.197.254:46655
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:39857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:08:56.889390 19033 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:08:56.889617 19033 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:08:56.889669 19033 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:08:56.891355 19033 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:08:56.891444 19033 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:08:56.891490 19033 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.193
I20260429 14:08:56.894245 19033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:39857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.197.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:46655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.193
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:08:56.894651 19033 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:08:56.894871 19033 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:08:56.897715 19039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:56.897797 19038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:56.897797 19041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:08:56.897979 19033 server_base.cc:1061] running on GCE node
I20260429 14:08:56.898329 19033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:08:56.898581 19033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:08:56.899798 19033 hybrid_clock.cc:648] HybridClock initialized: now 1777471736899776 us; error 42 us; skew 500 ppm
Apr 29 14:08:56 dist-test-slave-8n77 krb5kdc[18921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471736, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:08:56.902737 19033 init.cc:377] Logged in from keytab as kudu/127.2.197.193@KRBTEST.COM (short username kudu)
I20260429 14:08:56.903342 19033 webserver.cc:492] Webserver started at http://127.2.197.193:34001/ using document root <none> and password file <none>
I20260429 14:08:56.903554 19033 fs_manager.cc:362] Metadata directory not provided
I20260429 14:08:56.903617 19033 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:08:56.903726 19033 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:08:56.904702 19033 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/data/instance:
uuid: "1cedfa33539b4180a052efef6210d349"
format_stamp: "Formatted at 2026-04-29 14:08:56 on dist-test-slave-8n77"
server_key: "cfb446128f6c04e58ac71a9d2d12d2a5"
server_key_iv: "7ac92a475fdf7394296d091001b95770"
server_key_version: "encryptionkey@0"
I20260429 14:08:56.905045 19033 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance:
uuid: "1cedfa33539b4180a052efef6210d349"
format_stamp: "Formatted at 2026-04-29 14:08:56 on dist-test-slave-8n77"
server_key: "cfb446128f6c04e58ac71a9d2d12d2a5"
server_key_iv: "7ac92a475fdf7394296d091001b95770"
server_key_version: "encryptionkey@0"
I20260429 14:08:56.906347 19033 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260429 14:08:56.907372 19048 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:56.907728 19033 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:08:56.907821 19033 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/wal
uuid: "1cedfa33539b4180a052efef6210d349"
format_stamp: "Formatted at 2026-04-29 14:08:56 on dist-test-slave-8n77"
server_key: "cfb446128f6c04e58ac71a9d2d12d2a5"
server_key_iv: "7ac92a475fdf7394296d091001b95770"
server_key_version: "encryptionkey@0"
I20260429 14:08:56.907902 19033 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:08:56.932199 19033 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:08:56.934139 19033 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:08:56.934334 19033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:08:56.934614 19033 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:08:56.934933 19033 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:08:56.934989 19033 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:56.935024 19033 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:08:56.935070 19033 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:56.941303 19033 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.193:45833
I20260429 14:08:56.941335 19161 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.193:45833 every 8 connection(s)
I20260429 14:08:56.941694 19033 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
I20260429 14:08:56.945717 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 19033
I20260429 14:08:56.945837 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance
I20260429 14:08:56.946034 2839 external_mini_cluster.cc:1468] Setting key e59e6c38a5462ecfa0ed30b70738f88f
Apr 29 14:08:56 dist-test-slave-8n77 krb5kdc[18921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471736, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:08:56.950762 19008 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:56.942536 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:41121 (local address 127.2.197.254:46655)
0429 14:08:56.942604 (+ 68us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:56.942608 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:56.942782 (+ 174us) server_negotiation.cc:408] Connection header received
0429 14:08:56.943237 (+ 455us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:56.943238 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:56.943248 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:56.943313 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:56.944048 (+ 735us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:56.944560 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:56.945153 (+ 593us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:56.945284 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:56.947954 (+ 2670us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:56.947958 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:56.947958 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:56.947965 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:56.949409 (+ 1444us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:56.949836 (+ 427us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:56.949838 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:56.949838 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:56.949865 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:56.950072 (+ 207us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:56.950073 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:56.950074 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:56.950176 (+ 102us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:56.950252 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:56.950549 (+ 297us) server_negotiation.cc:300] Negotiation successful
0429 14:08:56.950662 (+ 113us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260429 14:08:56.950873 19164 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:56.942516 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.193:41121)
0429 14:08:56.942668 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:56.942702 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:56.943162 (+ 460us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:56.943369 (+ 207us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:56.943372 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:56.943702 (+ 330us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:08:56.943954 (+ 252us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:56.943957 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:56.944668 (+ 711us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:56.944669 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:56.945069 (+ 400us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:56.945071 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:56.945172 (+ 101us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:56.946403 (+ 1231us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:08:56.946412 (+ 9us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:08:56.947849 (+ 1437us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:08:56.949522 (+ 1673us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:56.949530 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:56.949533 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:56.949757 (+ 224us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:56.949938 (+ 181us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:56.949939 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:56.949939 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:56.950007 (+ 68us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:56.950242 (+ 235us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:08:56.950246 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:08:56.950414 (+ 168us) client_negotiation.cc:770] Sending connection context
0429 14:08:56.950531 (+ 117us) client_negotiation.cc:241] Negotiation successful
0429 14:08:56.950675 (+ 144us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260429 14:08:56.951411 19162 heartbeater.cc:344] Connected to a master server at 127.2.197.254:46655
I20260429 14:08:56.951541 19162 heartbeater.cc:461] Registering TS with master...
I20260429 14:08:56.951776 19162 heartbeater.cc:507] Master 127.2.197.254:46655 requested a full tablet report, sending...
I20260429 14:08:56.952396 18969 ts_manager.cc:194] Registered new tserver with Master: 1cedfa33539b4180a052efef6210d349 (127.2.197.193:45833)
I20260429 14:08:56.953080 18969 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.193@KRBTEST.COM'} at 127.2.197.193:41121
WARNING: no policy specified for kudu/127.2.197.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:08:57.002104 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.194:0
--local_ip_for_outbound_sockets=127.2.197.194
--webserver_interface=127.2.197.194
--webserver_port=0
--tserver_master_addrs=127.2.197.254:46655
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:39857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:08:57.079754 19169 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:08:57.079938 19169 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:08:57.079963 19169 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:08:57.081408 19169 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:08:57.081452 19169 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:08:57.081477 19169 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.194
I20260429 14:08:57.083127 19169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:39857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.197.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:46655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.194
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:08:57.083309 19169 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:08:57.083449 19169 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:08:57.086055 19177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:57.086061 19175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:08:57.086089 19169 server_base.cc:1061] running on GCE node
W20260429 14:08:57.086061 19174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:08:57.086504 19169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:08:57.086781 19169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:08:57.088029 19169 hybrid_clock.cc:648] HybridClock initialized: now 1777471737088007 us; error 30 us; skew 500 ppm
Apr 29 14:08:57 dist-test-slave-8n77 krb5kdc[18921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471737, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:08:57.090575 19169 init.cc:377] Logged in from keytab as kudu/127.2.197.194@KRBTEST.COM (short username kudu)
I20260429 14:08:57.091127 19169 webserver.cc:492] Webserver started at http://127.2.197.194:43419/ using document root <none> and password file <none>
I20260429 14:08:57.091327 19169 fs_manager.cc:362] Metadata directory not provided
I20260429 14:08:57.091395 19169 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:08:57.091500 19169 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:08:57.092422 19169 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/data/instance:
uuid: "151984abb40245de8bd371bfa9754712"
format_stamp: "Formatted at 2026-04-29 14:08:57 on dist-test-slave-8n77"
server_key: "8261b44928bd71816fdb05dee8bf85ca"
server_key_iv: "73ae5cad3f4a5be6c4d3ad4c2df66412"
server_key_version: "encryptionkey@0"
I20260429 14:08:57.092784 19169 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance:
uuid: "151984abb40245de8bd371bfa9754712"
format_stamp: "Formatted at 2026-04-29 14:08:57 on dist-test-slave-8n77"
server_key: "8261b44928bd71816fdb05dee8bf85ca"
server_key_iv: "73ae5cad3f4a5be6c4d3ad4c2df66412"
server_key_version: "encryptionkey@0"
I20260429 14:08:57.094023 19169 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260429 14:08:57.095000 19184 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:57.095252 19169 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:08:57.095336 19169 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/wal
uuid: "151984abb40245de8bd371bfa9754712"
format_stamp: "Formatted at 2026-04-29 14:08:57 on dist-test-slave-8n77"
server_key: "8261b44928bd71816fdb05dee8bf85ca"
server_key_iv: "73ae5cad3f4a5be6c4d3ad4c2df66412"
server_key_version: "encryptionkey@0"
I20260429 14:08:57.095424 19169 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:08:57.111294 19169 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:08:57.113189 19169 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:08:57.113349 19169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:08:57.113618 19169 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:08:57.113934 19169 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:08:57.113991 19169 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:57.114032 19169 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:08:57.114061 19169 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:57.119943 19169 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.194:38431
I20260429 14:08:57.120033 19297 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.194:38431 every 8 connection(s)
I20260429 14:08:57.120328 19169 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
Apr 29 14:08:57 dist-test-slave-8n77 krb5kdc[18921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471737, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:08:57.127372 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 19169
I20260429 14:08:57.127514 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance
I20260429 14:08:57.127765 2839 external_mini_cluster.cc:1468] Setting key a84b9e6302975bab45f12ff4c295afe0
I20260429 14:08:57.129098 19300 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:57.121133 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.194:57599)
0429 14:08:57.121267 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:57.121301 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:57.121768 (+ 467us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:57.121992 (+ 224us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:57.121995 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:57.122322 (+ 327us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:08:57.122636 (+ 314us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:57.122640 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:57.123282 (+ 642us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:57.123283 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:57.123667 (+ 384us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:57.123668 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:57.123752 (+ 84us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:57.124614 (+ 862us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:08:57.124623 (+ 9us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:08:57.125992 (+ 1369us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:08:57.127792 (+ 1800us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:57.127798 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:57.127800 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:57.128042 (+ 242us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:57.128236 (+ 194us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:57.128237 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:57.128237 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:57.128308 (+ 71us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:57.128621 (+ 313us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:08:57.128627 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:08:57.128756 (+ 129us) client_negotiation.cc:770] Sending connection context
0429 14:08:57.128853 (+ 97us) client_negotiation.cc:241] Negotiation successful
0429 14:08:57.128956 (+ 103us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":78,"threads_started":1}
I20260429 14:08:57.129124 19008 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:57.121131 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:57599 (local address 127.2.197.254:46655)
0429 14:08:57.121199 (+ 68us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:57.121201 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:57.121416 (+ 215us) server_negotiation.cc:408] Connection header received
0429 14:08:57.121847 (+ 431us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:57.121848 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:57.121859 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:57.121939 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:57.122723 (+ 784us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:57.123207 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:57.123746 (+ 539us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:57.123864 (+ 118us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:57.126076 (+ 2212us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:57.126080 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:57.126081 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:57.126087 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:57.127701 (+ 1614us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:57.128135 (+ 434us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:57.128137 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:57.128137 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:57.128166 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:57.128424 (+ 258us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:57.128425 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:57.128426 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:57.128538 (+ 112us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:57.128590 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:57.128900 (+ 310us) server_negotiation.cc:300] Negotiation successful
0429 14:08:57.128990 (+ 90us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260429 14:08:57.129592 19298 heartbeater.cc:344] Connected to a master server at 127.2.197.254:46655
I20260429 14:08:57.129683 19298 heartbeater.cc:461] Registering TS with master...
I20260429 14:08:57.129858 19298 heartbeater.cc:507] Master 127.2.197.254:46655 requested a full tablet report, sending...
I20260429 14:08:57.130316 18969 ts_manager.cc:194] Registered new tserver with Master: 151984abb40245de8bd371bfa9754712 (127.2.197.194:38431)
I20260429 14:08:57.130715 18969 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.194@KRBTEST.COM'} at 127.2.197.194:57599
WARNING: no policy specified for kudu/127.2.197.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:08:57.182448 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.195:0
--local_ip_for_outbound_sockets=127.2.197.195
--webserver_interface=127.2.197.195
--webserver_port=0
--tserver_master_addrs=127.2.197.254:46655
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:39857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:08:57.260465 19305 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:08:57.260646 19305 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:08:57.260671 19305 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:08:57.262100 19305 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:08:57.262145 19305 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:08:57.262173 19305 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.195
I20260429 14:08:57.263911 19305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:39857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.197.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:46655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.195
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:08:57.264094 19305 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:08:57.264230 19305 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:08:57.266703 19311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:57.266703 19313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:08:57.266798 19310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:08:57.267086 19305 server_base.cc:1061] running on GCE node
I20260429 14:08:57.267277 19305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:08:57.267524 19305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:08:57.268692 19305 hybrid_clock.cc:648] HybridClock initialized: now 1777471737268678 us; error 32 us; skew 500 ppm
Apr 29 14:08:57 dist-test-slave-8n77 krb5kdc[18921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471737, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:08:57.271301 19305 init.cc:377] Logged in from keytab as kudu/127.2.197.195@KRBTEST.COM (short username kudu)
I20260429 14:08:57.271843 19305 webserver.cc:492] Webserver started at http://127.2.197.195:41087/ using document root <none> and password file <none>
I20260429 14:08:57.272045 19305 fs_manager.cc:362] Metadata directory not provided
I20260429 14:08:57.272089 19305 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:08:57.272188 19305 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:08:57.273164 19305 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/data/instance:
uuid: "848b695a07f8402b839455990e0514bc"
format_stamp: "Formatted at 2026-04-29 14:08:57 on dist-test-slave-8n77"
server_key: "24e77462343480f84a99ab4a8db83d00"
server_key_iv: "0ac3c7c301aee6eff4466efdef3ad877"
server_key_version: "encryptionkey@0"
I20260429 14:08:57.273485 19305 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance:
uuid: "848b695a07f8402b839455990e0514bc"
format_stamp: "Formatted at 2026-04-29 14:08:57 on dist-test-slave-8n77"
server_key: "24e77462343480f84a99ab4a8db83d00"
server_key_iv: "0ac3c7c301aee6eff4466efdef3ad877"
server_key_version: "encryptionkey@0"
I20260429 14:08:57.274780 19305 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260429 14:08:57.275571 19320 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:57.275987 19305 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:08:57.276049 19305 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/wal
uuid: "848b695a07f8402b839455990e0514bc"
format_stamp: "Formatted at 2026-04-29 14:08:57 on dist-test-slave-8n77"
server_key: "24e77462343480f84a99ab4a8db83d00"
server_key_iv: "0ac3c7c301aee6eff4466efdef3ad877"
server_key_version: "encryptionkey@0"
I20260429 14:08:57.276125 19305 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:08:57.297093 19305 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:08:57.299162 19305 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:08:57.299597 19305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:08:57.299852 19305 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:08:57.300165 19305 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:08:57.300218 19305 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:57.300275 19305 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:08:57.300333 19305 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:08:57.306990 19305 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.195:35315
I20260429 14:08:57.307029 19433 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.195:35315 every 8 connection(s)
I20260429 14:08:57.307368 19305 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
Apr 29 14:08:57 dist-test-slave-8n77 krb5kdc[18921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471737, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:08:57.316210 19008 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:57.308156 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:58455 (local address 127.2.197.254:46655)
0429 14:08:57.308275 (+ 119us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:57.308278 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:57.308447 (+ 169us) server_negotiation.cc:408] Connection header received
0429 14:08:57.308982 (+ 535us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:57.308983 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:57.308997 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:57.309059 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:57.309788 (+ 729us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:57.310426 (+ 638us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:57.311066 (+ 640us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:57.311252 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:57.313513 (+ 2261us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:57.313518 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:57.313519 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:57.313529 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:57.314928 (+ 1399us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:57.315343 (+ 415us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:57.315345 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:57.315345 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:57.315377 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:57.315578 (+ 201us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:57.315579 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:57.315580 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:57.315687 (+ 107us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:57.315766 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:57.316012 (+ 246us) server_negotiation.cc:300] Negotiation successful
0429 14:08:57.316110 (+ 98us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260429 14:08:57.316339 19436 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:57.308119 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.195:58455)
0429 14:08:57.308302 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:57.308349 (+ 47us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:57.308886 (+ 537us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:57.309123 (+ 237us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:57.309126 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:57.309427 (+ 301us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:08:57.309692 (+ 265us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:57.309695 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:57.310511 (+ 816us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:57.310512 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:57.310976 (+ 464us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:57.310978 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:57.311101 (+ 123us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:57.311997 (+ 896us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:08:57.312005 (+ 8us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:08:57.313421 (+ 1416us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:08:57.315035 (+ 1614us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:57.315041 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:57.315044 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:57.315251 (+ 207us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:57.315452 (+ 201us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:08:57.315452 (+ 0us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:08:57.315453 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:08:57.315515 (+ 62us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:08:57.315755 (+ 240us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:08:57.315760 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:08:57.315883 (+ 123us) client_negotiation.cc:770] Sending connection context
0429 14:08:57.315995 (+ 112us) client_negotiation.cc:241] Negotiation successful
0429 14:08:57.316133 (+ 138us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":77,"threads_started":1}
I20260429 14:08:57.317003 19434 heartbeater.cc:344] Connected to a master server at 127.2.197.254:46655
I20260429 14:08:57.317121 19434 heartbeater.cc:461] Registering TS with master...
I20260429 14:08:57.317369 19434 heartbeater.cc:507] Master 127.2.197.254:46655 requested a full tablet report, sending...
I20260429 14:08:57.317585 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 19305
I20260429 14:08:57.317700 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance
I20260429 14:08:57.317791 18969 ts_manager.cc:194] Registered new tserver with Master: 848b695a07f8402b839455990e0514bc (127.2.197.195:35315)
I20260429 14:08:57.317921 2839 external_mini_cluster.cc:1468] Setting key 0ecd5e481e1eaad260b38160a792172a
I20260429 14:08:57.318347 18969 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.195@KRBTEST.COM'} at 127.2.197.195:58455
I20260429 14:08:57.318869 2839 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260429 14:08:57.326184 19008 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:57.320269 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:59244 (local address 127.2.197.254:46655)
0429 14:08:57.320316 (+ 47us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:57.320319 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:57.320558 (+ 239us) server_negotiation.cc:408] Connection header received
0429 14:08:57.320582 (+ 24us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:57.320583 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:57.320592 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:57.320659 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:08:57.321359 (+ 700us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:57.321819 (+ 460us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:57.322443 (+ 624us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:57.322651 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:57.323520 (+ 869us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:08:57.323523 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:08:57.323524 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:08:57.323530 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:08:57.324988 (+ 1458us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:57.325315 (+ 327us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:57.325316 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:57.325316 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:57.325335 (+ 19us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:08:57.325508 (+ 173us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:08:57.325510 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:08:57.325510 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:08:57.325642 (+ 132us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:08:57.325760 (+ 118us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:57.325924 (+ 164us) server_negotiation.cc:300] Negotiation successful
0429 14:08:57.325997 (+ 73us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260429 14:08:57.328063 18969 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 14:08:57.329412 18969 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 14:08:57.334956 19446 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:57.331850 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:45833 (local address 127.0.0.1:38242)
0429 14:08:57.332047 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:57.332058 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:57.332118 (+ 60us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:57.332436 (+ 318us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:57.332439 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:57.332443 (+ 4us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:08:57.332690 (+ 247us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:57.332692 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:57.333817 (+ 1125us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:57.333818 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:57.334668 (+ 850us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:57.334670 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:57.334757 (+ 87us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:57.334793 (+ 36us) client_negotiation.cc:770] Sending connection context
0429 14:08:57.334839 (+ 46us) client_negotiation.cc:241] Negotiation successful
0429 14:08:57.334879 (+ 40us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":90,"threads_started":1}
I20260429 14:08:57.335057 19445 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:57.331775 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:38431 (local address 127.0.0.1:52664)
0429 14:08:57.331918 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:57.331954 (+ 36us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:57.332034 (+ 80us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:57.332522 (+ 488us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:57.332524 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:57.332527 (+ 3us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:08:57.332777 (+ 250us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:57.332779 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:57.334169 (+ 1390us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:57.334170 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:57.334817 (+ 647us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:57.334819 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:57.334911 (+ 92us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:57.334915 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 14:08:57.334949 (+ 34us) client_negotiation.cc:241] Negotiation successful
0429 14:08:57.334988 (+ 39us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260429 14:08:57.335562 19448 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:57.331989 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:35315 (local address 127.0.0.1:42356)
0429 14:08:57.332190 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:57.332199 (+ 9us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:57.332247 (+ 48us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:57.332604 (+ 357us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:57.332605 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:57.332607 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:08:57.332787 (+ 180us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:57.332788 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:57.334129 (+ 1341us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:57.334130 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:57.335361 (+ 1231us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:57.335363 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:57.335426 (+ 63us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:57.335430 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 14:08:57.335462 (+ 32us) client_negotiation.cc:241] Negotiation successful
0429 14:08:57.335503 (+ 41us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":63,"threads_started":1}
I20260429 14:08:57.335803 19447 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:57.331878 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:52664 (local address 127.2.197.194:38431)
0429 14:08:57.332321 (+ 443us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:57.332329 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:57.332340 (+ 11us) server_negotiation.cc:408] Connection header received
0429 14:08:57.332381 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:57.332383 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:57.332399 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:57.332856 (+ 457us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:08:57.332949 (+ 93us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:57.334110 (+ 1161us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:57.334935 (+ 825us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:57.335579 (+ 644us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:57.335651 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:57.335684 (+ 33us) server_negotiation.cc:300] Negotiation successful
0429 14:08:57.335728 (+ 44us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":95,"threads_started":1}
I20260429 14:08:57.336051 19449 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:57.332090 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:38242 (local address 127.2.197.193:45833)
0429 14:08:57.332277 (+ 187us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:57.332287 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:57.332296 (+ 9us) server_negotiation.cc:408] Connection header received
0429 14:08:57.332336 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:57.332337 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:57.332350 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:57.332426 (+ 76us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:08:57.332771 (+ 345us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:57.333729 (+ 958us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:57.335303 (+ 1574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:57.335867 (+ 564us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:57.335907 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:57.335935 (+ 28us) server_negotiation.cc:300] Negotiation successful
0429 14:08:57.335976 (+ 41us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":111,"threads_started":1}
I20260429 14:08:57.336378 19450 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:57.332205 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42356 (local address 127.2.197.195:35315)
0429 14:08:57.332354 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:57.332360 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:57.332368 (+ 8us) server_negotiation.cc:408] Connection header received
0429 14:08:57.332440 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:57.332443 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:57.332457 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:57.332522 (+ 65us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:08:57.332869 (+ 347us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:57.334040 (+ 1171us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:57.335747 (+ 1707us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:57.336242 (+ 495us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:57.336268 (+ 26us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:57.336292 (+ 24us) server_negotiation.cc:300] Negotiation successful
0429 14:08:57.336320 (+ 28us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":69,"threads_started":1}
I20260429 14:08:57.336313 19232 tablet_service.cc:1511] Processing CreateTablet for tablet 76ce97852e524ec297dd1f1d148bcb04 (DEFAULT_TABLE table=test-table [id=10b08effaf86489b90e02b3f06303661]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:08:57.336599 19232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76ce97852e524ec297dd1f1d148bcb04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:08:57.336826 19368 tablet_service.cc:1511] Processing CreateTablet for tablet 76ce97852e524ec297dd1f1d148bcb04 (DEFAULT_TABLE table=test-table [id=10b08effaf86489b90e02b3f06303661]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:08:57.337037 19368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76ce97852e524ec297dd1f1d148bcb04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:08:57.338742 19096 tablet_service.cc:1511] Processing CreateTablet for tablet 76ce97852e524ec297dd1f1d148bcb04 (DEFAULT_TABLE table=test-table [id=10b08effaf86489b90e02b3f06303661]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:08:57.339004 19096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76ce97852e524ec297dd1f1d148bcb04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:08:57.339143 19451 tablet_bootstrap.cc:492] T 76ce97852e524ec297dd1f1d148bcb04 P 848b695a07f8402b839455990e0514bc: Bootstrap starting.
I20260429 14:08:57.339412 19452 tablet_bootstrap.cc:492] T 76ce97852e524ec297dd1f1d148bcb04 P 151984abb40245de8bd371bfa9754712: Bootstrap starting.
I20260429 14:08:57.339890 19451 tablet_bootstrap.cc:654] T 76ce97852e524ec297dd1f1d148bcb04 P 848b695a07f8402b839455990e0514bc: Neither blocks nor log segments found. Creating new log.
I20260429 14:08:57.339986 19452 tablet_bootstrap.cc:654] T 76ce97852e524ec297dd1f1d148bcb04 P 151984abb40245de8bd371bfa9754712: Neither blocks nor log segments found. Creating new log.
I20260429 14:08:57.340224 19451 log.cc:826] T 76ce97852e524ec297dd1f1d148bcb04 P 848b695a07f8402b839455990e0514bc: Log is configured to *not* fsync() on all Append() calls
I20260429 14:08:57.340266 19452 log.cc:826] T 76ce97852e524ec297dd1f1d148bcb04 P 151984abb40245de8bd371bfa9754712: Log is configured to *not* fsync() on all Append() calls
I20260429 14:08:57.341040 19451 tablet_bootstrap.cc:492] T 76ce97852e524ec297dd1f1d148bcb04 P 848b695a07f8402b839455990e0514bc: No bootstrap required, opened a new log
I20260429 14:08:57.341115 19451 ts_tablet_manager.cc:1403] T 76ce97852e524ec297dd1f1d148bcb04 P 848b695a07f8402b839455990e0514bc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260429 14:08:57.341228 19452 tablet_bootstrap.cc:492] T 76ce97852e524ec297dd1f1d148bcb04 P 151984abb40245de8bd371bfa9754712: No bootstrap required, opened a new log
I20260429 14:08:57.341308 19452 ts_tablet_manager.cc:1403] T 76ce97852e524ec297dd1f1d148bcb04 P 151984abb40245de8bd371bfa9754712: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260429 14:08:57.341620 19455 tablet_bootstrap.cc:492] T 76ce97852e524ec297dd1f1d148bcb04 P 1cedfa33539b4180a052efef6210d349: Bootstrap starting.
I20260429 14:08:57.342226 19455 tablet_bootstrap.cc:654] T 76ce97852e524ec297dd1f1d148bcb04 P 1cedfa33539b4180a052efef6210d349: Neither blocks nor log segments found. Creating new log.
I20260429 14:08:57.342562 19455 log.cc:826] T 76ce97852e524ec297dd1f1d148bcb04 P 1cedfa33539b4180a052efef6210d349: Log is configured to *not* fsync() on all Append() calls
I20260429 14:08:57.342679 19451 raft_consensus.cc:359] T 76ce97852e524ec297dd1f1d148bcb04 P 848b695a07f8402b839455990e0514bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151984abb40245de8bd371bfa9754712" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38431 } } peers { permanent_uuid: "848b695a07f8402b839455990e0514bc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35315 } } peers { permanent_uuid: "1cedfa33539b4180a052efef6210d349" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45833 } }
I20260429 14:08:57.342852 19451 raft_consensus.cc:385] T 76ce97852e524ec297dd1f1d148bcb04 P 848b695a07f8402b839455990e0514bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:08:57.342901 19451 raft_consensus.cc:740] T 76ce97852e524ec297dd1f1d148bcb04 P 848b695a07f8402b839455990e0514bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 848b695a07f8402b839455990e0514bc, State: Initialized, Role: FOLLOWER
I20260429 14:08:57.342998 19451 consensus_queue.cc:260] T 76ce97852e524ec297dd1f1d148bcb04 P 848b695a07f8402b839455990e0514bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151984abb40245de8bd371bfa9754712" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38431 } } peers { permanent_uuid: "848b695a07f8402b839455990e0514bc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35315 } } peers { permanent_uuid: "1cedfa33539b4180a052efef6210d349" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45833 } }
I20260429 14:08:57.343261 19451 ts_tablet_manager.cc:1434] T 76ce97852e524ec297dd1f1d148bcb04 P 848b695a07f8402b839455990e0514bc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260429 14:08:57.343319 19455 tablet_bootstrap.cc:492] T 76ce97852e524ec297dd1f1d148bcb04 P 1cedfa33539b4180a052efef6210d349: No bootstrap required, opened a new log
I20260429 14:08:57.343442 19455 ts_tablet_manager.cc:1403] T 76ce97852e524ec297dd1f1d148bcb04 P 1cedfa33539b4180a052efef6210d349: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260429 14:08:57.343320 19434 heartbeater.cc:499] Master 127.2.197.254:46655 was elected leader, sending a full tablet report...
I20260429 14:08:57.343757 19452 raft_consensus.cc:359] T 76ce97852e524ec297dd1f1d148bcb04 P 151984abb40245de8bd371bfa9754712 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151984abb40245de8bd371bfa9754712" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38431 } } peers { permanent_uuid: "848b695a07f8402b839455990e0514bc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35315 } } peers { permanent_uuid: "1cedfa33539b4180a052efef6210d349" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45833 } }
I20260429 14:08:57.343981 19452 raft_consensus.cc:385] T 76ce97852e524ec297dd1f1d148bcb04 P 151984abb40245de8bd371bfa9754712 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:08:57.344028 19452 raft_consensus.cc:740] T 76ce97852e524ec297dd1f1d148bcb04 P 151984abb40245de8bd371bfa9754712 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 151984abb40245de8bd371bfa9754712, State: Initialized, Role: FOLLOWER
I20260429 14:08:57.344115 19452 consensus_queue.cc:260] T 76ce97852e524ec297dd1f1d148bcb04 P 151984abb40245de8bd371bfa9754712 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151984abb40245de8bd371bfa9754712" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38431 } } peers { permanent_uuid: "848b695a07f8402b839455990e0514bc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35315 } } peers { permanent_uuid: "1cedfa33539b4180a052efef6210d349" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45833 } }
I20260429 14:08:57.344399 19452 ts_tablet_manager.cc:1434] T 76ce97852e524ec297dd1f1d148bcb04 P 151984abb40245de8bd371bfa9754712: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260429 14:08:57.344511 19298 heartbeater.cc:499] Master 127.2.197.254:46655 was elected leader, sending a full tablet report...
I20260429 14:08:57.345786 19455 raft_consensus.cc:359] T 76ce97852e524ec297dd1f1d148bcb04 P 1cedfa33539b4180a052efef6210d349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151984abb40245de8bd371bfa9754712" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38431 } } peers { permanent_uuid: "848b695a07f8402b839455990e0514bc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35315 } } peers { permanent_uuid: "1cedfa33539b4180a052efef6210d349" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45833 } }
I20260429 14:08:57.345909 19455 raft_consensus.cc:385] T 76ce97852e524ec297dd1f1d148bcb04 P 1cedfa33539b4180a052efef6210d349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:08:57.345929 19455 raft_consensus.cc:740] T 76ce97852e524ec297dd1f1d148bcb04 P 1cedfa33539b4180a052efef6210d349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cedfa33539b4180a052efef6210d349, State: Initialized, Role: FOLLOWER
I20260429 14:08:57.346041 19455 consensus_queue.cc:260] T 76ce97852e524ec297dd1f1d148bcb04 P 1cedfa33539b4180a052efef6210d349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151984abb40245de8bd371bfa9754712" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38431 } } peers { permanent_uuid: "848b695a07f8402b839455990e0514bc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35315 } } peers { permanent_uuid: "1cedfa33539b4180a052efef6210d349" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45833 } }
I20260429 14:08:57.346333 19455 ts_tablet_manager.cc:1434] T 76ce97852e524ec297dd1f1d148bcb04 P 1cedfa33539b4180a052efef6210d349: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260429 14:08:57.346422 19162 heartbeater.cc:499] Master 127.2.197.254:46655 was elected leader, sending a full tablet report...
W20260429 14:08:57.370887 19299 tablet.cc:2404] T 76ce97852e524ec297dd1f1d148bcb04 P 151984abb40245de8bd371bfa9754712: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 14:08:57.402556 19458 raft_consensus.cc:493] T 76ce97852e524ec297dd1f1d148bcb04 P 151984abb40245de8bd371bfa9754712 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:08:57.402690 19458 raft_consensus.cc:515] T 76ce97852e524ec297dd1f1d148bcb04 P 151984abb40245de8bd371bfa9754712 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151984abb40245de8bd371bfa9754712" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38431 } } peers { permanent_uuid: "848b695a07f8402b839455990e0514bc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35315 } } peers { permanent_uuid: "1cedfa33539b4180a052efef6210d349" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45833 } }
I20260429 14:08:57.402979 19458 leader_election.cc:290] T 76ce97852e524ec297dd1f1d148bcb04 P 151984abb40245de8bd371bfa9754712 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 848b695a07f8402b839455990e0514bc (127.2.197.195:35315), 1cedfa33539b4180a052efef6210d349 (127.2.197.193:45833)
I20260429 14:08:57.405354 19300 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:57.403213 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:35315 (local address 127.2.197.194:34093)
0429 14:08:57.403262 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:57.403274 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:57.403370 (+ 96us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:57.403543 (+ 173us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:57.403545 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:57.403547 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:08:57.403741 (+ 194us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:57.403742 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:57.404457 (+ 715us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:57.404458 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:57.405048 (+ 590us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:57.405049 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:57.405169 (+ 120us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:57.405174 (+ 5us) client_negotiation.cc:770] Sending connection context
0429 14:08:57.405210 (+ 36us) client_negotiation.cc:241] Negotiation successful
0429 14:08:57.405248 (+ 38us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260429 14:08:57.405465 19460 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:57.403213 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:45833 (local address 127.2.197.194:34481)
0429 14:08:57.403462 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:57.403475 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 14:08:57.403553 (+ 78us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:08:57.403720 (+ 167us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:08:57.403721 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:08:57.403724 (+ 3us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:08:57.403935 (+ 211us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:57.403937 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:57.404576 (+ 639us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:08:57.404577 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:08:57.405122 (+ 545us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:08:57.405124 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:08:57.405214 (+ 90us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:57.405217 (+ 3us) client_negotiation.cc:770] Sending connection context
0429 14:08:57.405249 (+ 32us) client_negotiation.cc:241] Negotiation successful
0429 14:08:57.405284 (+ 35us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":60,"thread_start_us":116,"threads_started":1}
I20260429 14:08:57.405778 19450 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:57.403292 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:34093 (local address 127.2.197.195:35315)
0429 14:08:57.403343 (+ 51us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:57.403345 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:57.403428 (+ 83us) server_negotiation.cc:408] Connection header received
0429 14:08:57.403454 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:57.403455 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:57.403463 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:57.403533 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:08:57.403843 (+ 310us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:57.404344 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:57.405141 (+ 797us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:57.405655 (+ 514us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:57.405667 (+ 12us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:57.405693 (+ 26us) server_negotiation.cc:300] Negotiation successful
0429 14:08:57.405733 (+ 40us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260429 14:08:57.405853 19449 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:57.403272 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:34481 (local address 127.2.197.193:45833)
0429 14:08:57.403332 (+ 60us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:57.403334 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:57.403579 (+ 245us) server_negotiation.cc:408] Connection header received
0429 14:08:57.403612 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:57.403613 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:57.403623 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:57.403690 (+ 67us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:08:57.404029 (+ 339us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:57.404486 (+ 457us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:57.405213 (+ 727us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:57.405729 (+ 516us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:57.405737 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:57.405759 (+ 22us) server_negotiation.cc:300] Negotiation successful
0429 14:08:57.405791 (+ 32us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260429 14:08:57.406154 19116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ce97852e524ec297dd1f1d148bcb04" candidate_uuid: "151984abb40245de8bd371bfa9754712" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cedfa33539b4180a052efef6210d349" is_pre_election: true
I20260429 14:08:57.406149 19388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ce97852e524ec297dd1f1d148bcb04" candidate_uuid: "151984abb40245de8bd371bfa9754712" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "848b695a07f8402b839455990e0514bc" is_pre_election: true
I20260429 14:08:57.406337 19116 raft_consensus.cc:2468] T 76ce97852e524ec297dd1f1d148bcb04 P 1cedfa33539b4180a052efef6210d349 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 151984abb40245de8bd371bfa9754712 in term 0.
I20260429 14:08:57.406351 19388 raft_consensus.cc:2468] T 76ce97852e524ec297dd1f1d148bcb04 P 848b695a07f8402b839455990e0514bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 151984abb40245de8bd371bfa9754712 in term 0.
I20260429 14:08:57.406639 19185 leader_election.cc:304] T 76ce97852e524ec297dd1f1d148bcb04 P 151984abb40245de8bd371bfa9754712 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 151984abb40245de8bd371bfa9754712, 848b695a07f8402b839455990e0514bc; no voters:
I20260429 14:08:57.406814 19458 raft_consensus.cc:2804] T 76ce97852e524ec297dd1f1d148bcb04 P 151984abb40245de8bd371bfa9754712 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 14:08:57.406879 19458 raft_consensus.cc:493] T 76ce97852e524ec297dd1f1d148bcb04 P 151984abb40245de8bd371bfa9754712 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 14:08:57.406905 19458 raft_consensus.cc:3060] T 76ce97852e524ec297dd1f1d148bcb04 P 151984abb40245de8bd371bfa9754712 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:08:57.407589 19458 raft_consensus.cc:515] T 76ce97852e524ec297dd1f1d148bcb04 P 151984abb40245de8bd371bfa9754712 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151984abb40245de8bd371bfa9754712" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38431 } } peers { permanent_uuid: "848b695a07f8402b839455990e0514bc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35315 } } peers { permanent_uuid: "1cedfa33539b4180a052efef6210d349" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45833 } }
I20260429 14:08:57.407747 19458 leader_election.cc:290] T 76ce97852e524ec297dd1f1d148bcb04 P 151984abb40245de8bd371bfa9754712 [CANDIDATE]: Term 1 election: Requested vote from peers 848b695a07f8402b839455990e0514bc (127.2.197.195:35315), 1cedfa33539b4180a052efef6210d349 (127.2.197.193:45833)
I20260429 14:08:57.407938 19388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ce97852e524ec297dd1f1d148bcb04" candidate_uuid: "151984abb40245de8bd371bfa9754712" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "848b695a07f8402b839455990e0514bc"
I20260429 14:08:57.408008 19388 raft_consensus.cc:3060] T 76ce97852e524ec297dd1f1d148bcb04 P 848b695a07f8402b839455990e0514bc [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:08:57.408125 19116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ce97852e524ec297dd1f1d148bcb04" candidate_uuid: "151984abb40245de8bd371bfa9754712" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cedfa33539b4180a052efef6210d349"
I20260429 14:08:57.408209 19116 raft_consensus.cc:3060] T 76ce97852e524ec297dd1f1d148bcb04 P 1cedfa33539b4180a052efef6210d349 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:08:57.408737 19388 raft_consensus.cc:2468] T 76ce97852e524ec297dd1f1d148bcb04 P 848b695a07f8402b839455990e0514bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 151984abb40245de8bd371bfa9754712 in term 1.
I20260429 14:08:57.408921 19116 raft_consensus.cc:2468] T 76ce97852e524ec297dd1f1d148bcb04 P 1cedfa33539b4180a052efef6210d349 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 151984abb40245de8bd371bfa9754712 in term 1.
I20260429 14:08:57.408921 19185 leader_election.cc:304] T 76ce97852e524ec297dd1f1d148bcb04 P 151984abb40245de8bd371bfa9754712 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 151984abb40245de8bd371bfa9754712, 848b695a07f8402b839455990e0514bc; no voters:
I20260429 14:08:57.409036 19458 raft_consensus.cc:2804] T 76ce97852e524ec297dd1f1d148bcb04 P 151984abb40245de8bd371bfa9754712 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:08:57.409220 19458 raft_consensus.cc:697] T 76ce97852e524ec297dd1f1d148bcb04 P 151984abb40245de8bd371bfa9754712 [term 1 LEADER]: Becoming Leader. State: Replica: 151984abb40245de8bd371bfa9754712, State: Running, Role: LEADER
I20260429 14:08:57.409337 19458 consensus_queue.cc:237] T 76ce97852e524ec297dd1f1d148bcb04 P 151984abb40245de8bd371bfa9754712 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151984abb40245de8bd371bfa9754712" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38431 } } peers { permanent_uuid: "848b695a07f8402b839455990e0514bc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35315 } } peers { permanent_uuid: "1cedfa33539b4180a052efef6210d349" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45833 } }
I20260429 14:08:57.410156 18969 catalog_manager.cc:5671] T 76ce97852e524ec297dd1f1d148bcb04 P 151984abb40245de8bd371bfa9754712 reported cstate change: term changed from 0 to 1, leader changed from <none> to 151984abb40245de8bd371bfa9754712 (127.2.197.194). New cstate: current_term: 1 leader_uuid: "151984abb40245de8bd371bfa9754712" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151984abb40245de8bd371bfa9754712" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "848b695a07f8402b839455990e0514bc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cedfa33539b4180a052efef6210d349" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45833 } health_report { overall_health: UNKNOWN } } }
I20260429 14:08:57.430202 19447 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:57.428002 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:52672 (local address 127.2.197.194:38431)
0429 14:08:57.428069 (+ 67us) server_negotiation.cc:207] Beginning negotiation
0429 14:08:57.428071 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:08:57.428084 (+ 13us) server_negotiation.cc:408] Connection header received
0429 14:08:57.428137 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:08:57.428138 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:08:57.428160 (+ 22us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:08:57.428230 (+ 70us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:08:57.428503 (+ 273us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:57.428978 (+ 475us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:08:57.429636 (+ 658us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:08:57.429806 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:08:57.429840 (+ 34us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:08:57.429940 (+ 100us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:08:57.430033 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0429 14:08:57.430087 (+ 54us) server_negotiation.cc:300] Negotiation successful
0429 14:08:57.430111 (+ 24us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260429 14:08:57.432008 19388 raft_consensus.cc:1275] T 76ce97852e524ec297dd1f1d148bcb04 P 848b695a07f8402b839455990e0514bc [term 1 FOLLOWER]: Refusing update from remote peer 151984abb40245de8bd371bfa9754712: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:08:57.432034 19116 raft_consensus.cc:1275] T 76ce97852e524ec297dd1f1d148bcb04 P 1cedfa33539b4180a052efef6210d349 [term 1 FOLLOWER]: Refusing update from remote peer 151984abb40245de8bd371bfa9754712: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:08:57.432453 19458 consensus_queue.cc:1048] T 76ce97852e524ec297dd1f1d148bcb04 P 151984abb40245de8bd371bfa9754712 [LEADER]: Connected to new peer: Peer: permanent_uuid: "848b695a07f8402b839455990e0514bc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:08:57.432590 19461 consensus_queue.cc:1048] T 76ce97852e524ec297dd1f1d148bcb04 P 151984abb40245de8bd371bfa9754712 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cedfa33539b4180a052efef6210d349" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:08:57.434727 19464 mvcc.cc:204] Tried to move back new op lower bound from 7280524236518350848 to 7280524236429262848. Current Snapshot: MvccSnapshot[applied={T|T < 7280524236518350848}]
I20260429 14:08:57.437316 18969 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:08:57.437426 18969 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:08:57.438498 18969 catalog_manager.cc:5958] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22: Sending DeleteTablet for 3 replicas of tablet 76ce97852e524ec297dd1f1d148bcb04
I20260429 14:08:57.438855 19232 tablet_service.cc:1558] Processing DeleteTablet for tablet 76ce97852e524ec297dd1f1d148bcb04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:08:57 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:52664
I20260429 14:08:57.438920 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 19033
I20260429 14:08:57.439400 19368 tablet_service.cc:1558] Processing DeleteTablet for tablet 76ce97852e524ec297dd1f1d148bcb04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:08:57 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:42356
I20260429 14:08:57.439546 19474 tablet_replica.cc:333] T 76ce97852e524ec297dd1f1d148bcb04 P 151984abb40245de8bd371bfa9754712: stopping tablet replica
I20260429 14:08:57.439646 19474 raft_consensus.cc:2243] T 76ce97852e524ec297dd1f1d148bcb04 P 151984abb40245de8bd371bfa9754712 [term 1 LEADER]: Raft consensus shutting down.
I20260429 14:08:57.439733 19475 tablet_replica.cc:333] T 76ce97852e524ec297dd1f1d148bcb04 P 848b695a07f8402b839455990e0514bc: stopping tablet replica
I20260429 14:08:57.439755 19474 raft_consensus.cc:2272] T 76ce97852e524ec297dd1f1d148bcb04 P 151984abb40245de8bd371bfa9754712 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:08:57.439869 19475 raft_consensus.cc:2243] T 76ce97852e524ec297dd1f1d148bcb04 P 848b695a07f8402b839455990e0514bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:08:57.440011 19475 raft_consensus.cc:2272] T 76ce97852e524ec297dd1f1d148bcb04 P 848b695a07f8402b839455990e0514bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:08:57.440266 19474 ts_tablet_manager.cc:1916] T 76ce97852e524ec297dd1f1d148bcb04 P 151984abb40245de8bd371bfa9754712: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:08:57.440546 19475 ts_tablet_manager.cc:1916] T 76ce97852e524ec297dd1f1d148bcb04 P 848b695a07f8402b839455990e0514bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:08:57.442417 19475 ts_tablet_manager.cc:1929] T 76ce97852e524ec297dd1f1d148bcb04 P 848b695a07f8402b839455990e0514bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 14:08:57.442492 19474 ts_tablet_manager.cc:1929] T 76ce97852e524ec297dd1f1d148bcb04 P 151984abb40245de8bd371bfa9754712: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 14:08:57.442572 19474 log.cc:1199] T 76ce97852e524ec297dd1f1d148bcb04 P 151984abb40245de8bd371bfa9754712: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/wal/wals/76ce97852e524ec297dd1f1d148bcb04
I20260429 14:08:57.442862 19474 ts_tablet_manager.cc:1950] T 76ce97852e524ec297dd1f1d148bcb04 P 151984abb40245de8bd371bfa9754712: Deleting consensus metadata
I20260429 14:08:57.442945 19475 log.cc:1199] T 76ce97852e524ec297dd1f1d148bcb04 P 848b695a07f8402b839455990e0514bc: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/wal/wals/76ce97852e524ec297dd1f1d148bcb04
I20260429 14:08:57.443310 19475 ts_tablet_manager.cc:1950] T 76ce97852e524ec297dd1f1d148bcb04 P 848b695a07f8402b839455990e0514bc: Deleting consensus metadata
I20260429 14:08:57.443295 18953 catalog_manager.cc:5002] TS 151984abb40245de8bd371bfa9754712 (127.2.197.194:38431): tablet 76ce97852e524ec297dd1f1d148bcb04 (table test-table [id=10b08effaf86489b90e02b3f06303661]) successfully deleted
I20260429 14:08:57.443820 18953 catalog_manager.cc:5002] TS 848b695a07f8402b839455990e0514bc (127.2.197.195:35315): tablet 76ce97852e524ec297dd1f1d148bcb04 (table test-table [id=10b08effaf86489b90e02b3f06303661]) successfully deleted
W20260429 14:08:57.446779 18955 connection.cc:570] client connection to 127.2.197.193:45833 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260429 14:08:57.446854 18955 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260429 14:08:57.447021 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 19169
I20260429 14:08:57.448565 19448 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:08:57.448398 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:45833 (local address 127.0.0.1:38252)
0429 14:08:57.448476 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0429 14:08:57.448526 (+ 50us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.193:45833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260429 14:08:57.448747 18955 catalog_manager.cc:4729] TS 1cedfa33539b4180a052efef6210d349 (127.2.197.193:45833): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 76ce97852e524ec297dd1f1d148bcb04: Network error: Client connection negotiation failed: client connection to 127.2.197.193:45833: connect: Connection refused (error 111)
I20260429 14:08:57.452041 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 19305
I20260429 14:08:57.457141 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 18937
I20260429 14:09:17.462636 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:46655
--webserver_interface=127.2.197.254
--webserver_port=32977
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:39857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:46655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:09:17.540688 19477 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:09:17.540910 19477 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:09:17.540934 19477 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:09:17.542553 19477 flags.cc:432] Enabled experimental flag: --authn_token_validity_seconds=20
W20260429 14:09:17.542604 19477 flags.cc:432] Enabled experimental flag: --authz_token_validity_seconds=20
W20260429 14:09:17.542618 19477 flags.cc:432] Enabled experimental flag: --tsk_rotation_seconds=20
W20260429 14:09:17.542629 19477 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:09:17.542642 19477 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:09:17.542655 19477 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:09:17.542716 19477 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:09:17.542734 19477 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:09:17.544441 19477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:39857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:46655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:46655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=32977
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:09:17.544689 19477 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:09:17.544895 19477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:09:17.551617 19486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:09:17.551671 19477 server_base.cc:1061] running on GCE node
W20260429 14:09:17.551617 19483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:09:17.551648 19484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:09:17.552124 19477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:09:17.552412 19477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:09:17.553601 19477 hybrid_clock.cc:648] HybridClock initialized: now 1777471757553557 us; error 60 us; skew 500 ppm
Apr 29 14:09:17 dist-test-slave-8n77 krb5kdc[18921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471757, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:09:17.556556 19477 init.cc:377] Logged in from keytab as kudu/127.2.197.254@KRBTEST.COM (short username kudu)
I20260429 14:09:17.557188 19477 webserver.cc:492] Webserver started at http://127.2.197.254:32977/ using document root <none> and password file <none>
I20260429 14:09:17.557453 19477 fs_manager.cc:362] Metadata directory not provided
I20260429 14:09:17.557516 19477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:09:17.558946 19477 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260429 14:09:17.559619 19493 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:09:17.559834 19477 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:09:17.559921 19477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "a17d1d014a204c9b99cb95ef72965f22"
format_stamp: "Formatted at 2026-04-29 14:08:56 on dist-test-slave-8n77"
server_key: "8d0393de5cf257c663414fd2a24b5bda"
server_key_iv: "6ab51389eb13d6c82a55acf23ecdad1a"
server_key_version: "encryptionkey@0"
I20260429 14:09:17.560328 19477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:09:17.579385 19477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:09:17.581557 19477 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:09:17.581749 19477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:09:17.586032 19477 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:46655
I20260429 14:09:17.586104 19545 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:46655 every 8 connection(s)
I20260429 14:09:17.586540 19477 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:09:17.587576 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 19477
I20260429 14:09:17.587857 19546 sys_catalog.cc:263] Verifying existing consensus state
I20260429 14:09:17.587917 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.193:45833
--local_ip_for_outbound_sockets=127.2.197.193
--tserver_master_addrs=127.2.197.254:46655
--webserver_port=34001
--webserver_interface=127.2.197.193
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:39857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260429 14:09:17.588554 19546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22: Bootstrap starting.
I20260429 14:09:17.598774 19546 log.cc:826] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22: Log is configured to *not* fsync() on all Append() calls
I20260429 14:09:17.601047 19546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260429 14:09:17.601403 19546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22: Bootstrap complete.
I20260429 14:09:17.603525 19546 raft_consensus.cc:359] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a17d1d014a204c9b99cb95ef72965f22" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 46655 } }
I20260429 14:09:17.603710 19546 raft_consensus.cc:740] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a17d1d014a204c9b99cb95ef72965f22, State: Initialized, Role: FOLLOWER
I20260429 14:09:17.603858 19546 consensus_queue.cc:260] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a17d1d014a204c9b99cb95ef72965f22" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 46655 } }
I20260429 14:09:17.603928 19546 raft_consensus.cc:399] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:09:17.603981 19546 raft_consensus.cc:493] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:09:17.604029 19546 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [term 1 FOLLOWER]: Advancing to term 2
I20260429 14:09:17.604892 19546 raft_consensus.cc:515] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a17d1d014a204c9b99cb95ef72965f22" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 46655 } }
I20260429 14:09:17.604988 19546 leader_election.cc:304] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a17d1d014a204c9b99cb95ef72965f22; no voters:
I20260429 14:09:17.605221 19546 leader_election.cc:290] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260429 14:09:17.605263 19550 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [term 2 FOLLOWER]: Leader election won for term 2
I20260429 14:09:17.605422 19550 raft_consensus.cc:697] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [term 2 LEADER]: Becoming Leader. State: Replica: a17d1d014a204c9b99cb95ef72965f22, State: Running, Role: LEADER
I20260429 14:09:17.605469 19546 sys_catalog.cc:565] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:09:17.605582 19550 consensus_queue.cc:237] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a17d1d014a204c9b99cb95ef72965f22" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 46655 } }
I20260429 14:09:17.605937 19551 sys_catalog.cc:455] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a17d1d014a204c9b99cb95ef72965f22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a17d1d014a204c9b99cb95ef72965f22" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 46655 } } }
I20260429 14:09:17.606005 19551 sys_catalog.cc:458] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [sys.catalog]: This master's current role is: LEADER
I20260429 14:09:17.606212 19552 sys_catalog.cc:455] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a17d1d014a204c9b99cb95ef72965f22. Latest consensus state: current_term: 2 leader_uuid: "a17d1d014a204c9b99cb95ef72965f22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a17d1d014a204c9b99cb95ef72965f22" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 46655 } } }
I20260429 14:09:17.606305 19552 sys_catalog.cc:458] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [sys.catalog]: This master's current role is: LEADER
I20260429 14:09:17.606495 19555 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:09:17.607239 19555 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:09:17.607426 19555 catalog_manager.cc:1269] Loaded cluster ID: 97b25306dfc84bfcbe5e5f50aca04bc0
I20260429 14:09:17.607604 19555 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:09:17.608408 19555 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:09:17.608527 19555 catalog_manager.cc:6055] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22: Loaded TSK: 0
I20260429 14:09:17.620106 19555 catalog_manager.cc:6044] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22: Generated new TSK 1
I20260429 14:09:17.620275 19555 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260429 14:09:17.678493 19548 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:09:17.678700 19548 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:09:17.678726 19548 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:09:17.680377 19548 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:09:17.680429 19548 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:09:17.680459 19548 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.193
I20260429 14:09:17.682142 19548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:39857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.193:45833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.197.193
--webserver_port=34001
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:46655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.193
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:09:17.682415 19548 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:09:17.682624 19548 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:09:17.685101 19573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:09:17.685107 19576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:09:17.685213 19574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:09:17.685441 19548 server_base.cc:1061] running on GCE node
I20260429 14:09:17.685648 19548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:09:17.685889 19548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:09:17.687063 19548 hybrid_clock.cc:648] HybridClock initialized: now 1777471757687048 us; error 27 us; skew 500 ppm
Apr 29 14:09:17 dist-test-slave-8n77 krb5kdc[18921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471757, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:09:17.689862 19548 init.cc:377] Logged in from keytab as kudu/127.2.197.193@KRBTEST.COM (short username kudu)
I20260429 14:09:17.690564 19548 webserver.cc:492] Webserver started at http://127.2.197.193:34001/ using document root <none> and password file <none>
I20260429 14:09:17.690798 19548 fs_manager.cc:362] Metadata directory not provided
I20260429 14:09:17.690872 19548 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:09:17.692163 19548 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:09:17.692977 19583 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:09:17.693207 19548 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:09:17.693782 19548 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/wal
uuid: "1cedfa33539b4180a052efef6210d349"
format_stamp: "Formatted at 2026-04-29 14:08:56 on dist-test-slave-8n77"
server_key: "cfb446128f6c04e58ac71a9d2d12d2a5"
server_key_iv: "7ac92a475fdf7394296d091001b95770"
server_key_version: "encryptionkey@0"
I20260429 14:09:17.694099 19548 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:09:17.716243 19548 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:09:17.718724 19548 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:09:17.718889 19548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:09:17.719128 19548 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:09:17.719594 19590 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260429 14:09:17.720758 19548 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260429 14:09:17.720831 19548 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260429 14:09:17.720880 19548 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260429 14:09:17.721488 19548 ts_tablet_manager.cc:616] Registered 1 tablets
I20260429 14:09:17.721573 19548 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260429 14:09:17.721604 19590 tablet_bootstrap.cc:492] T 76ce97852e524ec297dd1f1d148bcb04 P 1cedfa33539b4180a052efef6210d349: Bootstrap starting.
I20260429 14:09:17.729421 19548 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.193:45833
I20260429 14:09:17.729815 19548 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
I20260429 14:09:17.733847 19590 log.cc:826] T 76ce97852e524ec297dd1f1d148bcb04 P 1cedfa33539b4180a052efef6210d349: Log is configured to *not* fsync() on all Append() calls
I20260429 14:09:17.734045 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 19548
I20260429 14:09:17.734464 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.194:38431
--local_ip_for_outbound_sockets=127.2.197.194
--tserver_master_addrs=127.2.197.254:46655
--webserver_port=43419
--webserver_interface=127.2.197.194
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:39857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260429 14:09:17.735569 19697 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.193:45833 every 8 connection(s)
I20260429 14:09:17.736307 19590 tablet_bootstrap.cc:492] T 76ce97852e524ec297dd1f1d148bcb04 P 1cedfa33539b4180a052efef6210d349: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260429 14:09:17.736709 19590 tablet_bootstrap.cc:492] T 76ce97852e524ec297dd1f1d148bcb04 P 1cedfa33539b4180a052efef6210d349: Bootstrap complete.
I20260429 14:09:17.738035 19590 ts_tablet_manager.cc:1403] T 76ce97852e524ec297dd1f1d148bcb04 P 1cedfa33539b4180a052efef6210d349: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.011s
I20260429 14:09:17.739280 19590 raft_consensus.cc:359] T 76ce97852e524ec297dd1f1d148bcb04 P 1cedfa33539b4180a052efef6210d349 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151984abb40245de8bd371bfa9754712" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38431 } } peers { permanent_uuid: "848b695a07f8402b839455990e0514bc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35315 } } peers { permanent_uuid: "1cedfa33539b4180a052efef6210d349" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45833 } }
I20260429 14:09:17.739660 19590 raft_consensus.cc:740] T 76ce97852e524ec297dd1f1d148bcb04 P 1cedfa33539b4180a052efef6210d349 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cedfa33539b4180a052efef6210d349, State: Initialized, Role: FOLLOWER
I20260429 14:09:17.740185 19590 consensus_queue.cc:260] T 76ce97852e524ec297dd1f1d148bcb04 P 1cedfa33539b4180a052efef6210d349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151984abb40245de8bd371bfa9754712" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38431 } } peers { permanent_uuid: "848b695a07f8402b839455990e0514bc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35315 } } peers { permanent_uuid: "1cedfa33539b4180a052efef6210d349" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45833 } }
I20260429 14:09:17.740531 19590 ts_tablet_manager.cc:1434] T 76ce97852e524ec297dd1f1d148bcb04 P 1cedfa33539b4180a052efef6210d349: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
Apr 29 14:09:17 dist-test-slave-8n77 krb5kdc[18921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471757, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:09:17.746901 19700 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:17.730738 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.193:52979)
0429 14:09:17.734471 (+ 3733us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:17.734532 (+ 61us) client_negotiation.cc:175] Beginning negotiation
0429 14:09:17.735293 (+ 761us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:09:17.736333 (+ 1040us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:09:17.736338 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:09:17.736810 (+ 472us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:09:17.737149 (+ 339us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:17.737153 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:17.738740 (+ 1587us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:09:17.738741 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:09:17.739440 (+ 699us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:17.739443 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:17.739577 (+ 134us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:17.740365 (+ 788us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:09:17.740378 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:09:17.742310 (+ 1932us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:09:17.744997 (+ 2687us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:09:17.745008 (+ 11us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:09:17.745011 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:09:17.745333 (+ 322us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:09:17.745638 (+ 305us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:09:17.745639 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:09:17.745639 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:09:17.745731 (+ 92us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:09:17.746235 (+ 504us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:09:17.746241 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:09:17.746415 (+ 174us) client_negotiation.cc:770] Sending connection context
0429 14:09:17.746533 (+ 118us) client_negotiation.cc:241] Negotiation successful
0429 14:09:17.746704 (+ 171us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3676,"thread_start_us":87,"threads_started":1}
I20260429 14:09:17.747169 19701 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:17.730784 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:52979 (local address 127.2.197.254:46655)
0429 14:09:17.735858 (+ 5074us) server_negotiation.cc:207] Beginning negotiation
0429 14:09:17.735871 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0429 14:09:17.735903 (+ 32us) server_negotiation.cc:408] Connection header received
0429 14:09:17.736024 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:09:17.736029 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:09:17.736079 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:09:17.736205 (+ 126us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:09:17.737288 (+ 1083us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:17.738622 (+ 1334us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:09:17.739874 (+ 1252us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:17.740187 (+ 313us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:17.742433 (+ 2246us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:09:17.742441 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:09:17.742442 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:09:17.742458 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:09:17.744864 (+ 2406us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:09:17.745452 (+ 588us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:09:17.745456 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:09:17.745457 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:09:17.745541 (+ 84us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:09:17.745886 (+ 345us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:09:17.745887 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:09:17.745888 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:09:17.746146 (+ 258us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:09:17.746251 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0429 14:09:17.746751 (+ 500us) server_negotiation.cc:300] Negotiation successful
0429 14:09:17.746904 (+ 153us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5024,"thread_start_us":73,"threads_started":1}
I20260429 14:09:17.748131 19698 heartbeater.cc:344] Connected to a master server at 127.2.197.254:46655
I20260429 14:09:17.748242 19698 heartbeater.cc:461] Registering TS with master...
I20260429 14:09:17.748483 19698 heartbeater.cc:507] Master 127.2.197.254:46655 requested a full tablet report, sending...
I20260429 14:09:17.749397 19510 ts_manager.cc:194] Registered new tserver with Master: 1cedfa33539b4180a052efef6210d349 (127.2.197.193:45833)
I20260429 14:09:17.750759 19510 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.193@KRBTEST.COM'} at 127.2.197.193:52979
I20260429 14:09:17.751689 19698 heartbeater.cc:499] Master 127.2.197.254:46655 was elected leader, sending a full tablet report...
I20260429 14:09:17.759358 19705 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:17.750081 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:45833 (local address 127.0.0.1:55268)
0429 14:09:17.754503 (+ 4422us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:17.754539 (+ 36us) client_negotiation.cc:175] Beginning negotiation
0429 14:09:17.754639 (+ 100us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:09:17.754841 (+ 202us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:09:17.754844 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:09:17.754847 (+ 3us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:09:17.755167 (+ 320us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:17.755171 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:17.757743 (+ 2572us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:09:17.757748 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:09:17.759020 (+ 1272us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:17.759029 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:17.759142 (+ 113us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:17.759174 (+ 32us) client_negotiation.cc:770] Sending connection context
0429 14:09:17.759221 (+ 47us) client_negotiation.cc:241] Negotiation successful
0429 14:09:17.759261 (+ 40us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4387,"thread_start_us":111,"threads_started":1}
I20260429 14:09:17.759955 19706 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:17.751042 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55268 (local address 127.2.197.193:45833)
0429 14:09:17.751205 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0429 14:09:17.751214 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0429 14:09:17.754638 (+ 3424us) server_negotiation.cc:408] Connection header received
0429 14:09:17.754714 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:09:17.754717 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:09:17.754735 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:09:17.754823 (+ 88us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:09:17.755293 (+ 470us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:17.757140 (+ 1847us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:09:17.759179 (+ 2039us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:17.759756 (+ 577us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:17.759799 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0429 14:09:17.759832 (+ 33us) server_negotiation.cc:300] Negotiation successful
0429 14:09:17.759874 (+ 42us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
I20260429 14:09:17.760251 19632 tablet_service.cc:1558] Processing DeleteTablet for tablet 76ce97852e524ec297dd1f1d148bcb04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:08:57 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:55268
I20260429 14:09:17.760843 19707 tablet_replica.cc:333] stopping tablet replica
I20260429 14:09:17.760943 19707 raft_consensus.cc:2243] T 76ce97852e524ec297dd1f1d148bcb04 P 1cedfa33539b4180a052efef6210d349 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:09:17.761063 19707 raft_consensus.cc:2272] T 76ce97852e524ec297dd1f1d148bcb04 P 1cedfa33539b4180a052efef6210d349 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:09:17.761445 19707 ts_tablet_manager.cc:1916] T 76ce97852e524ec297dd1f1d148bcb04 P 1cedfa33539b4180a052efef6210d349: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:09:17.764160 19707 ts_tablet_manager.cc:1929] T 76ce97852e524ec297dd1f1d148bcb04 P 1cedfa33539b4180a052efef6210d349: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 14:09:17.764246 19707 log.cc:1199] T 76ce97852e524ec297dd1f1d148bcb04 P 1cedfa33539b4180a052efef6210d349: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/wal/wals/76ce97852e524ec297dd1f1d148bcb04
I20260429 14:09:17.764520 19707 ts_tablet_manager.cc:1950] T 76ce97852e524ec297dd1f1d148bcb04 P 1cedfa33539b4180a052efef6210d349: Deleting consensus metadata
I20260429 14:09:17.765091 19496 catalog_manager.cc:5002] TS 1cedfa33539b4180a052efef6210d349 (127.2.197.193:45833): tablet 76ce97852e524ec297dd1f1d148bcb04 (table test-table [id=10b08effaf86489b90e02b3f06303661]) successfully deleted
W20260429 14:09:17.826848 19703 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:09:17.827032 19703 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:09:17.827056 19703 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:09:17.828567 19703 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:09:17.828612 19703 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:09:17.828639 19703 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.194
I20260429 14:09:17.830163 19703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:39857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.194:38431
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.197.194
--webserver_port=43419
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:46655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.194
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:09:17.830422 19703 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:09:17.830561 19703 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:09:17.833150 19715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:09:17.833230 19712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:09:17.833307 19703 server_base.cc:1061] running on GCE node
W20260429 14:09:17.833230 19713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:09:17.833778 19703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:09:17.834080 19703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:09:17.835261 19703 hybrid_clock.cc:648] HybridClock initialized: now 1777471757835238 us; error 41 us; skew 500 ppm
Apr 29 14:09:17 dist-test-slave-8n77 krb5kdc[18921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471757, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:09:17.838028 19703 init.cc:377] Logged in from keytab as kudu/127.2.197.194@KRBTEST.COM (short username kudu)
I20260429 14:09:17.838757 19703 webserver.cc:492] Webserver started at http://127.2.197.194:43419/ using document root <none> and password file <none>
I20260429 14:09:17.838965 19703 fs_manager.cc:362] Metadata directory not provided
I20260429 14:09:17.839038 19703 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:09:17.840391 19703 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:09:17.841089 19722 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:09:17.841306 19703 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:09:17.841427 19703 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/wal
uuid: "151984abb40245de8bd371bfa9754712"
format_stamp: "Formatted at 2026-04-29 14:08:57 on dist-test-slave-8n77"
server_key: "8261b44928bd71816fdb05dee8bf85ca"
server_key_iv: "73ae5cad3f4a5be6c4d3ad4c2df66412"
server_key_version: "encryptionkey@0"
I20260429 14:09:17.841729 19703 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:09:17.869755 19703 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:09:17.872141 19703 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:09:17.872359 19703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:09:17.872617 19703 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:09:17.872928 19703 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:09:17.872982 19703 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:09:17.873021 19703 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:09:17.873063 19703 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:09:17.880821 19703 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.194:38431
I20260429 14:09:17.881022 19835 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.194:38431 every 8 connection(s)
I20260429 14:09:17.881419 19703 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
Apr 29 14:09:17 dist-test-slave-8n77 krb5kdc[18921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471757, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:09:17.891286 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 19703
I20260429 14:09:17.891665 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.195:35315
--local_ip_for_outbound_sockets=127.2.197.195
--tserver_master_addrs=127.2.197.254:46655
--webserver_port=41087
--webserver_interface=127.2.197.195
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:39857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260429 14:09:17.893265 19701 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:17.882479 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:38543 (local address 127.2.197.254:46655)
0429 14:09:17.882696 (+ 217us) server_negotiation.cc:207] Beginning negotiation
0429 14:09:17.882698 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:09:17.882954 (+ 256us) server_negotiation.cc:408] Connection header received
0429 14:09:17.883744 (+ 790us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:09:17.883746 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:09:17.883764 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:09:17.883825 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:09:17.884911 (+ 1086us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:17.886072 (+ 1161us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:09:17.886699 (+ 627us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:17.886830 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:17.889587 (+ 2757us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:09:17.889598 (+ 11us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:09:17.889598 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:09:17.889607 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:09:17.891373 (+ 1766us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:09:17.891946 (+ 573us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:09:17.891948 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:09:17.891948 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:09:17.891978 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:09:17.892395 (+ 417us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:09:17.892396 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:09:17.892397 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:09:17.892540 (+ 143us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:09:17.892622 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0429 14:09:17.893064 (+ 442us) server_negotiation.cc:300] Negotiation successful
0429 14:09:17.893176 (+ 112us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186}
I20260429 14:09:17.893767 19838 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:17.882404 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.194:38543)
0429 14:09:17.882786 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:17.882847 (+ 61us) client_negotiation.cc:175] Beginning negotiation
0429 14:09:17.883631 (+ 784us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:09:17.883883 (+ 252us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:09:17.883887 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:09:17.884381 (+ 494us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:09:17.884766 (+ 385us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:17.884771 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:17.886155 (+ 1384us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:09:17.886156 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:09:17.886612 (+ 456us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:17.886614 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:17.886721 (+ 107us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:17.887330 (+ 609us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:09:17.887341 (+ 11us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:09:17.889450 (+ 2109us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:09:17.891498 (+ 2048us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:09:17.891507 (+ 9us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:09:17.891511 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:09:17.891860 (+ 349us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:09:17.892132 (+ 272us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:09:17.892135 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:09:17.892135 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:09:17.892277 (+ 142us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:09:17.892661 (+ 384us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:09:17.892669 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:09:17.892858 (+ 189us) client_negotiation.cc:770] Sending connection context
0429 14:09:17.893417 (+ 559us) client_negotiation.cc:241] Negotiation successful
0429 14:09:17.893601 (+ 184us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":207,"threads_started":1}
I20260429 14:09:17.894435 19836 heartbeater.cc:344] Connected to a master server at 127.2.197.254:46655
I20260429 14:09:17.894552 19836 heartbeater.cc:461] Registering TS with master...
I20260429 14:09:17.894788 19836 heartbeater.cc:507] Master 127.2.197.254:46655 requested a full tablet report, sending...
I20260429 14:09:17.895263 19510 ts_manager.cc:194] Registered new tserver with Master: 151984abb40245de8bd371bfa9754712 (127.2.197.194:38431)
I20260429 14:09:17.895871 19510 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.194@KRBTEST.COM'} at 127.2.197.194:38543
W20260429 14:09:17.971287 19839 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:09:17.971472 19839 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:09:17.971496 19839 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:09:17.973008 19839 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:09:17.973062 19839 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:09:17.973096 19839 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.195
I20260429 14:09:17.974834 19839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:39857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.195:35315
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.197.195
--webserver_port=41087
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:46655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.195
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:09:17.975078 19839 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:09:17.975281 19839 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260429 14:09:17.978128 19839 server_base.cc:1061] running on GCE node
W20260429 14:09:17.978102 19847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:09:17.978102 19844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:09:17.978435 19845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:09:17.978726 19839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:09:17.978960 19839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:09:17.980118 19839 hybrid_clock.cc:648] HybridClock initialized: now 1777471757980118 us; error 43 us; skew 500 ppm
Apr 29 14:09:17 dist-test-slave-8n77 krb5kdc[18921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471757, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:09:17.982483 19839 init.cc:377] Logged in from keytab as kudu/127.2.197.195@KRBTEST.COM (short username kudu)
I20260429 14:09:17.983049 19839 webserver.cc:492] Webserver started at http://127.2.197.195:41087/ using document root <none> and password file <none>
I20260429 14:09:17.983247 19839 fs_manager.cc:362] Metadata directory not provided
I20260429 14:09:17.983326 19839 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:09:17.984521 19839 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:09:17.985100 19854 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:09:17.985275 19839 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260429 14:09:17.985349 19839 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/wal
uuid: "848b695a07f8402b839455990e0514bc"
format_stamp: "Formatted at 2026-04-29 14:08:57 on dist-test-slave-8n77"
server_key: "24e77462343480f84a99ab4a8db83d00"
server_key_iv: "0ac3c7c301aee6eff4466efdef3ad877"
server_key_version: "encryptionkey@0"
I20260429 14:09:17.985651 19839 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:09:17.992921 19839 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:09:17.994670 19839 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:09:17.994812 19839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:09:17.995018 19839 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:09:17.995347 19839 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:09:17.995398 19839 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:09:17.995440 19839 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:09:17.995466 19839 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:09:18.001163 19839 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.195:35315
I20260429 14:09:18.001397 19967 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.195:35315 every 8 connection(s)
I20260429 14:09:18.001549 19839 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
I20260429 14:09:18.007045 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 19839
Apr 29 14:09:18 dist-test-slave-8n77 krb5kdc[18921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471757, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:09:18.010855 19701 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:18.002343 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:43355 (local address 127.2.197.254:46655)
0429 14:09:18.002405 (+ 62us) server_negotiation.cc:207] Beginning negotiation
0429 14:09:18.002407 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:09:18.002625 (+ 218us) server_negotiation.cc:408] Connection header received
0429 14:09:18.003075 (+ 450us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:09:18.003076 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:09:18.003087 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:09:18.003138 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:09:18.003810 (+ 672us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:18.004296 (+ 486us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:09:18.004852 (+ 556us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:18.004964 (+ 112us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:18.007868 (+ 2904us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:09:18.007872 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:09:18.007872 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:09:18.007880 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:09:18.009477 (+ 1597us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:09:18.009926 (+ 449us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:09:18.009928 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:09:18.009928 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:09:18.009957 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:09:18.010148 (+ 191us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:09:18.010149 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:09:18.010150 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:09:18.010248 (+ 98us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:09:18.010370 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
0429 14:09:18.010685 (+ 315us) server_negotiation.cc:300] Negotiation successful
0429 14:09:18.010790 (+ 105us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260429 14:09:18.011246 19970 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:18.002360 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.195:43355)
0429 14:09:18.002495 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:18.002529 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0429 14:09:18.002995 (+ 466us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:09:18.003176 (+ 181us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:09:18.003179 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:09:18.003471 (+ 292us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:09:18.003722 (+ 251us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:18.003725 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:18.004383 (+ 658us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:09:18.004384 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:09:18.004777 (+ 393us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:18.004779 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:18.004873 (+ 94us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:18.005312 (+ 439us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:09:18.005320 (+ 8us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:09:18.007774 (+ 2454us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:09:18.009571 (+ 1797us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:09:18.009577 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:09:18.009579 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:09:18.009835 (+ 256us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:09:18.010034 (+ 199us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:09:18.010034 (+ 0us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:09:18.010034 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:09:18.010089 (+ 55us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:09:18.010379 (+ 290us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:09:18.010389 (+ 10us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:09:18.010497 (+ 108us) client_negotiation.cc:770] Sending connection context
0429 14:09:18.010640 (+ 143us) client_negotiation.cc:241] Negotiation successful
0429 14:09:18.011048 (+ 408us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":72,"threads_started":1}
I20260429 14:09:18.011758 19968 heartbeater.cc:344] Connected to a master server at 127.2.197.254:46655
I20260429 14:09:18.011861 19968 heartbeater.cc:461] Registering TS with master...
I20260429 14:09:18.012058 19968 heartbeater.cc:507] Master 127.2.197.254:46655 requested a full tablet report, sending...
I20260429 14:09:18.012485 19510 ts_manager.cc:194] Registered new tserver with Master: 848b695a07f8402b839455990e0514bc (127.2.197.195:35315)
I20260429 14:09:18.013005 19510 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.195@KRBTEST.COM'} at 127.2.197.195:43355
I20260429 14:09:18.014806 19972 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:18.007635 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33774 (local address 127.2.197.254:46655)
0429 14:09:18.007762 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0429 14:09:18.007764 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:09:18.007895 (+ 131us) server_negotiation.cc:408] Connection header received
0429 14:09:18.008008 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:09:18.008009 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:09:18.008023 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:09:18.008092 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:09:18.009188 (+ 1096us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:18.009814 (+ 626us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:09:18.010581 (+ 767us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:18.010771 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:18.011929 (+ 1158us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:09:18.011933 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:09:18.011933 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:09:18.011940 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:09:18.013669 (+ 1729us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:09:18.014010 (+ 341us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:09:18.014010 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:09:18.014011 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:09:18.014029 (+ 18us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:09:18.014196 (+ 167us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:09:18.014196 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:09:18.014197 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:09:18.014267 (+ 70us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:09:18.014469 (+ 202us) server_negotiation.cc:1036] Waiting for connection context
0429 14:09:18.014618 (+ 149us) server_negotiation.cc:300] Negotiation successful
0429 14:09:18.014709 (+ 91us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":58,"threads_started":1}
I20260429 14:09:18.015542 2839 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260429 14:09:18.017800 19972 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:18.015875 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33778 (local address 127.2.197.254:46655)
0429 14:09:18.015925 (+ 50us) server_negotiation.cc:207] Beginning negotiation
0429 14:09:18.015927 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:09:18.016048 (+ 121us) server_negotiation.cc:408] Connection header received
0429 14:09:18.016093 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:09:18.016094 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:09:18.016112 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:09:18.016158 (+ 46us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:09:18.016372 (+ 214us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:18.016819 (+ 447us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:09:18.017437 (+ 618us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:18.017603 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:18.017632 (+ 29us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:09:18.017665 (+ 33us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0429 14:09:18.017766 (+ 101us) negotiation.cc:326] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33778: token expired
Metrics: {"server-negotiator.queue_time_us":30}
W20260429 14:09:18.017875 19972 negotiation.cc:343] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33778: token expired
W20260429 14:09:18.018005 19439 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.197.254:46655)
I20260429 14:09:18.023254 19972 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:18.018329 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33784 (local address 127.2.197.254:46655)
0429 14:09:18.018362 (+ 33us) server_negotiation.cc:207] Beginning negotiation
0429 14:09:18.018364 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:09:18.018371 (+ 7us) server_negotiation.cc:408] Connection header received
0429 14:09:18.018424 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:09:18.018425 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:09:18.018433 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:09:18.018477 (+ 44us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:09:18.018981 (+ 504us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:18.019419 (+ 438us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:09:18.019886 (+ 467us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:18.019986 (+ 100us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:18.020920 (+ 934us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:09:18.020924 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:09:18.020924 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:09:18.020930 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:09:18.022107 (+ 1177us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:09:18.022658 (+ 551us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:09:18.022659 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:09:18.022659 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:09:18.022680 (+ 21us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:09:18.022813 (+ 133us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:09:18.022814 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:09:18.022814 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:09:18.022904 (+ 90us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:09:18.022976 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
0429 14:09:18.023063 (+ 87us) server_negotiation.cc:300] Negotiation successful
0429 14:09:18.023129 (+ 66us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260429 14:09:18.024256 19510 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 14:09:18.025482 19510 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 14:09:18.027443 19632 tablet_service.cc:1511] Processing CreateTablet for tablet 8649b568deb04ed9aa157a3b4c8a20f1 (DEFAULT_TABLE table=test-table [id=2bbbcbbefb6a41d991e8abf3cd258d50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:09:18.027635 19632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8649b568deb04ed9aa157a3b4c8a20f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:09:18.029078 19590 tablet_bootstrap.cc:492] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349: Bootstrap starting.
I20260429 14:09:18.029433 19590 tablet_bootstrap.cc:654] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349: Neither blocks nor log segments found. Creating new log.
I20260429 14:09:18.030051 19590 tablet_bootstrap.cc:492] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349: No bootstrap required, opened a new log
I20260429 14:09:18.030109 19590 ts_tablet_manager.cc:1403] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260429 14:09:18.030225 19590 raft_consensus.cc:359] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848b695a07f8402b839455990e0514bc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35315 } } peers { permanent_uuid: "151984abb40245de8bd371bfa9754712" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38431 } } peers { permanent_uuid: "1cedfa33539b4180a052efef6210d349" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45833 } }
I20260429 14:09:18.030316 19590 raft_consensus.cc:385] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:09:18.030354 19590 raft_consensus.cc:740] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cedfa33539b4180a052efef6210d349, State: Initialized, Role: FOLLOWER
I20260429 14:09:18.030378 19705 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:18.027240 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:35315 (local address 127.0.0.1:48062)
0429 14:09:18.027282 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:18.027294 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:09:18.027328 (+ 34us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:09:18.027809 (+ 481us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:09:18.027810 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:09:18.027815 (+ 5us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:09:18.027991 (+ 176us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:18.027993 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:18.029360 (+ 1367us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:09:18.029361 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:09:18.030098 (+ 737us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:18.030100 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:18.030185 (+ 85us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:18.030189 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 14:09:18.030221 (+ 32us) client_negotiation.cc:241] Negotiation successful
0429 14:09:18.030260 (+ 39us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260429 14:09:18.030398 19590 consensus_queue.cc:260] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848b695a07f8402b839455990e0514bc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35315 } } peers { permanent_uuid: "151984abb40245de8bd371bfa9754712" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38431 } } peers { permanent_uuid: "1cedfa33539b4180a052efef6210d349" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45833 } }
I20260429 14:09:18.030467 19975 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:18.027338 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:38431 (local address 127.0.0.1:50918)
0429 14:09:18.027475 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:18.027485 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 14:09:18.027527 (+ 42us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:09:18.027962 (+ 435us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:09:18.027962 (+ 0us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:09:18.027964 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:09:18.028213 (+ 249us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:18.028216 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:18.029218 (+ 1002us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:09:18.029219 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:09:18.030161 (+ 942us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:18.030165 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:18.030228 (+ 63us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:18.030234 (+ 6us) client_negotiation.cc:770] Sending connection context
0429 14:09:18.030266 (+ 32us) client_negotiation.cc:241] Negotiation successful
0429 14:09:18.030339 (+ 73us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
I20260429 14:09:18.030488 19590 ts_tablet_manager.cc:1434] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260429 14:09:18.030937 19974 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:18.027306 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48062 (local address 127.2.197.195:35315)
0429 14:09:18.027652 (+ 346us) server_negotiation.cc:207] Beginning negotiation
0429 14:09:18.027660 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0429 14:09:18.027670 (+ 10us) server_negotiation.cc:408] Connection header received
0429 14:09:18.027707 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:09:18.027709 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:09:18.027723 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:09:18.027797 (+ 74us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:09:18.028077 (+ 280us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:18.029232 (+ 1155us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:09:18.030192 (+ 960us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:18.030733 (+ 541us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:18.030795 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0429 14:09:18.030828 (+ 33us) server_negotiation.cc:300] Negotiation successful
0429 14:09:18.030869 (+ 41us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":118,"threads_started":1}
I20260429 14:09:18.031163 19976 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:18.027570 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:50918 (local address 127.2.197.194:38431)
0429 14:09:18.027738 (+ 168us) server_negotiation.cc:207] Beginning negotiation
0429 14:09:18.027743 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 14:09:18.027752 (+ 9us) server_negotiation.cc:408] Connection header received
0429 14:09:18.027802 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:09:18.027804 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:09:18.027820 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:09:18.027890 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:09:18.028361 (+ 471us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:18.029123 (+ 762us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:09:18.030432 (+ 1309us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:18.030982 (+ 550us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:18.031022 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0429 14:09:18.031050 (+ 28us) server_negotiation.cc:300] Negotiation successful
0429 14:09:18.031087 (+ 37us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":103,"threads_started":1}
I20260429 14:09:18.031912 19902 tablet_service.cc:1511] Processing CreateTablet for tablet 8649b568deb04ed9aa157a3b4c8a20f1 (DEFAULT_TABLE table=test-table [id=2bbbcbbefb6a41d991e8abf3cd258d50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:09:18.031968 19752 tablet_service.cc:1511] Processing CreateTablet for tablet 8649b568deb04ed9aa157a3b4c8a20f1 (DEFAULT_TABLE table=test-table [id=2bbbcbbefb6a41d991e8abf3cd258d50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:09:18.032181 19752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8649b568deb04ed9aa157a3b4c8a20f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:09:18.032181 19902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8649b568deb04ed9aa157a3b4c8a20f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:09:18.036985 19978 tablet_bootstrap.cc:492] T 8649b568deb04ed9aa157a3b4c8a20f1 P 151984abb40245de8bd371bfa9754712: Bootstrap starting.
I20260429 14:09:18.036665 19979 tablet_bootstrap.cc:492] T 8649b568deb04ed9aa157a3b4c8a20f1 P 848b695a07f8402b839455990e0514bc: Bootstrap starting.
I20260429 14:09:18.038017 19979 tablet_bootstrap.cc:654] T 8649b568deb04ed9aa157a3b4c8a20f1 P 848b695a07f8402b839455990e0514bc: Neither blocks nor log segments found. Creating new log.
I20260429 14:09:18.038321 19979 log.cc:826] T 8649b568deb04ed9aa157a3b4c8a20f1 P 848b695a07f8402b839455990e0514bc: Log is configured to *not* fsync() on all Append() calls
I20260429 14:09:18.038609 19978 tablet_bootstrap.cc:654] T 8649b568deb04ed9aa157a3b4c8a20f1 P 151984abb40245de8bd371bfa9754712: Neither blocks nor log segments found. Creating new log.
I20260429 14:09:18.038869 19978 log.cc:826] T 8649b568deb04ed9aa157a3b4c8a20f1 P 151984abb40245de8bd371bfa9754712: Log is configured to *not* fsync() on all Append() calls
I20260429 14:09:18.039083 19979 tablet_bootstrap.cc:492] T 8649b568deb04ed9aa157a3b4c8a20f1 P 848b695a07f8402b839455990e0514bc: No bootstrap required, opened a new log
I20260429 14:09:18.039158 19979 ts_tablet_manager.cc:1403] T 8649b568deb04ed9aa157a3b4c8a20f1 P 848b695a07f8402b839455990e0514bc: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260429 14:09:18.039559 19978 tablet_bootstrap.cc:492] T 8649b568deb04ed9aa157a3b4c8a20f1 P 151984abb40245de8bd371bfa9754712: No bootstrap required, opened a new log
I20260429 14:09:18.039609 19978 ts_tablet_manager.cc:1403] T 8649b568deb04ed9aa157a3b4c8a20f1 P 151984abb40245de8bd371bfa9754712: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260429 14:09:18.040520 19979 raft_consensus.cc:359] T 8649b568deb04ed9aa157a3b4c8a20f1 P 848b695a07f8402b839455990e0514bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848b695a07f8402b839455990e0514bc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35315 } } peers { permanent_uuid: "151984abb40245de8bd371bfa9754712" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38431 } } peers { permanent_uuid: "1cedfa33539b4180a052efef6210d349" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45833 } }
I20260429 14:09:18.040695 19979 raft_consensus.cc:385] T 8649b568deb04ed9aa157a3b4c8a20f1 P 848b695a07f8402b839455990e0514bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:09:18.040745 19979 raft_consensus.cc:740] T 8649b568deb04ed9aa157a3b4c8a20f1 P 848b695a07f8402b839455990e0514bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 848b695a07f8402b839455990e0514bc, State: Initialized, Role: FOLLOWER
I20260429 14:09:18.040841 19979 consensus_queue.cc:260] T 8649b568deb04ed9aa157a3b4c8a20f1 P 848b695a07f8402b839455990e0514bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848b695a07f8402b839455990e0514bc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35315 } } peers { permanent_uuid: "151984abb40245de8bd371bfa9754712" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38431 } } peers { permanent_uuid: "1cedfa33539b4180a052efef6210d349" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45833 } }
I20260429 14:09:18.041088 19979 ts_tablet_manager.cc:1434] T 8649b568deb04ed9aa157a3b4c8a20f1 P 848b695a07f8402b839455990e0514bc: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260429 14:09:18.041149 19968 heartbeater.cc:499] Master 127.2.197.254:46655 was elected leader, sending a full tablet report...
I20260429 14:09:18.041616 19978 raft_consensus.cc:359] T 8649b568deb04ed9aa157a3b4c8a20f1 P 151984abb40245de8bd371bfa9754712 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848b695a07f8402b839455990e0514bc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35315 } } peers { permanent_uuid: "151984abb40245de8bd371bfa9754712" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38431 } } peers { permanent_uuid: "1cedfa33539b4180a052efef6210d349" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45833 } }
I20260429 14:09:18.041744 19978 raft_consensus.cc:385] T 8649b568deb04ed9aa157a3b4c8a20f1 P 151984abb40245de8bd371bfa9754712 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:09:18.041775 19978 raft_consensus.cc:740] T 8649b568deb04ed9aa157a3b4c8a20f1 P 151984abb40245de8bd371bfa9754712 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 151984abb40245de8bd371bfa9754712, State: Initialized, Role: FOLLOWER
I20260429 14:09:18.041855 19978 consensus_queue.cc:260] T 8649b568deb04ed9aa157a3b4c8a20f1 P 151984abb40245de8bd371bfa9754712 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848b695a07f8402b839455990e0514bc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35315 } } peers { permanent_uuid: "151984abb40245de8bd371bfa9754712" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38431 } } peers { permanent_uuid: "1cedfa33539b4180a052efef6210d349" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45833 } }
I20260429 14:09:18.042073 19978 ts_tablet_manager.cc:1434] T 8649b568deb04ed9aa157a3b4c8a20f1 P 151984abb40245de8bd371bfa9754712: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260429 14:09:18.042193 19836 heartbeater.cc:499] Master 127.2.197.254:46655 was elected leader, sending a full tablet report...
W20260429 14:09:18.132086 19837 tablet.cc:2404] T 8649b568deb04ed9aa157a3b4c8a20f1 P 151984abb40245de8bd371bfa9754712: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260429 14:09:18.236129 19699 tablet.cc:2404] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260429 14:09:18.252107 19969 tablet.cc:2404] T 8649b568deb04ed9aa157a3b4c8a20f1 P 848b695a07f8402b839455990e0514bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 14:09:18.262578 19983 raft_consensus.cc:493] T 8649b568deb04ed9aa157a3b4c8a20f1 P 151984abb40245de8bd371bfa9754712 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:09:18.262722 19983 raft_consensus.cc:515] T 8649b568deb04ed9aa157a3b4c8a20f1 P 151984abb40245de8bd371bfa9754712 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848b695a07f8402b839455990e0514bc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35315 } } peers { permanent_uuid: "151984abb40245de8bd371bfa9754712" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38431 } } peers { permanent_uuid: "1cedfa33539b4180a052efef6210d349" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45833 } }
I20260429 14:09:18.263132 19983 leader_election.cc:290] T 8649b568deb04ed9aa157a3b4c8a20f1 P 151984abb40245de8bd371bfa9754712 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 848b695a07f8402b839455990e0514bc (127.2.197.195:35315), 1cedfa33539b4180a052efef6210d349 (127.2.197.193:45833)
I20260429 14:09:18.265993 19838 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:18.263352 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:35315 (local address 127.2.197.194:33401)
0429 14:09:18.263411 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:18.263423 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:09:18.263482 (+ 59us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:09:18.263646 (+ 164us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:09:18.263648 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:09:18.263650 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:09:18.263858 (+ 208us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:18.263859 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:18.264878 (+ 1019us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:09:18.264878 (+ 0us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:09:18.265760 (+ 882us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:18.265762 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:18.265836 (+ 74us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:18.265840 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 14:09:18.265880 (+ 40us) client_negotiation.cc:241] Negotiation successful
0429 14:09:18.265917 (+ 37us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260429 14:09:18.266189 19984 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:18.263399 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:45833 (local address 127.2.197.194:43517)
0429 14:09:18.263578 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:18.263585 (+ 7us) client_negotiation.cc:175] Beginning negotiation
0429 14:09:18.263666 (+ 81us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:09:18.264064 (+ 398us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:09:18.264065 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:09:18.264067 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:09:18.264265 (+ 198us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:18.264266 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:18.265155 (+ 889us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:09:18.265156 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:09:18.265971 (+ 815us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:18.265973 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:18.266037 (+ 64us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:18.266041 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 14:09:18.266073 (+ 32us) client_negotiation.cc:241] Negotiation successful
0429 14:09:18.266104 (+ 31us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":91,"threads_started":1}
I20260429 14:09:18.266584 19974 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:18.263432 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:33401 (local address 127.2.197.195:35315)
0429 14:09:18.263483 (+ 51us) server_negotiation.cc:207] Beginning negotiation
0429 14:09:18.263485 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:09:18.263496 (+ 11us) server_negotiation.cc:408] Connection header received
0429 14:09:18.263535 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:09:18.263536 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:09:18.263546 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:09:18.263629 (+ 83us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:09:18.263965 (+ 336us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:18.264792 (+ 827us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:09:18.265870 (+ 1078us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:18.266450 (+ 580us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:18.266460 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
0429 14:09:18.266486 (+ 26us) server_negotiation.cc:300] Negotiation successful
0429 14:09:18.266522 (+ 36us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260429 14:09:18.266721 19985 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:18.263493 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:43517 (local address 127.2.197.193:45833)
0429 14:09:18.263623 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0429 14:09:18.263625 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:09:18.263658 (+ 33us) server_negotiation.cc:408] Connection header received
0429 14:09:18.263913 (+ 255us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:09:18.263915 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:09:18.263934 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:09:18.264052 (+ 118us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:09:18.264359 (+ 307us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:18.265083 (+ 724us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:09:18.266073 (+ 990us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:18.266581 (+ 508us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:18.266589 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0429 14:09:18.266613 (+ 24us) server_negotiation.cc:300] Negotiation successful
0429 14:09:18.266652 (+ 39us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":62,"threads_started":1}
I20260429 14:09:18.266974 19922 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8649b568deb04ed9aa157a3b4c8a20f1" candidate_uuid: "151984abb40245de8bd371bfa9754712" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "848b695a07f8402b839455990e0514bc" is_pre_election: true
I20260429 14:09:18.266978 19646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8649b568deb04ed9aa157a3b4c8a20f1" candidate_uuid: "151984abb40245de8bd371bfa9754712" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cedfa33539b4180a052efef6210d349" is_pre_election: true
I20260429 14:09:18.267091 19922 raft_consensus.cc:2468] T 8649b568deb04ed9aa157a3b4c8a20f1 P 848b695a07f8402b839455990e0514bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 151984abb40245de8bd371bfa9754712 in term 0.
I20260429 14:09:18.267149 19646 raft_consensus.cc:2468] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 151984abb40245de8bd371bfa9754712 in term 0.
I20260429 14:09:18.267352 19723 leader_election.cc:304] T 8649b568deb04ed9aa157a3b4c8a20f1 P 151984abb40245de8bd371bfa9754712 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 151984abb40245de8bd371bfa9754712, 848b695a07f8402b839455990e0514bc; no voters:
I20260429 14:09:18.267495 19983 raft_consensus.cc:2804] T 8649b568deb04ed9aa157a3b4c8a20f1 P 151984abb40245de8bd371bfa9754712 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 14:09:18.267577 19983 raft_consensus.cc:493] T 8649b568deb04ed9aa157a3b4c8a20f1 P 151984abb40245de8bd371bfa9754712 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 14:09:18.267613 19983 raft_consensus.cc:3060] T 8649b568deb04ed9aa157a3b4c8a20f1 P 151984abb40245de8bd371bfa9754712 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:09:18.268251 19983 raft_consensus.cc:515] T 8649b568deb04ed9aa157a3b4c8a20f1 P 151984abb40245de8bd371bfa9754712 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848b695a07f8402b839455990e0514bc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35315 } } peers { permanent_uuid: "151984abb40245de8bd371bfa9754712" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38431 } } peers { permanent_uuid: "1cedfa33539b4180a052efef6210d349" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45833 } }
I20260429 14:09:18.268430 19983 leader_election.cc:290] T 8649b568deb04ed9aa157a3b4c8a20f1 P 151984abb40245de8bd371bfa9754712 [CANDIDATE]: Term 1 election: Requested vote from peers 848b695a07f8402b839455990e0514bc (127.2.197.195:35315), 1cedfa33539b4180a052efef6210d349 (127.2.197.193:45833)
I20260429 14:09:18.268574 19922 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8649b568deb04ed9aa157a3b4c8a20f1" candidate_uuid: "151984abb40245de8bd371bfa9754712" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "848b695a07f8402b839455990e0514bc"
I20260429 14:09:18.268657 19922 raft_consensus.cc:3060] T 8649b568deb04ed9aa157a3b4c8a20f1 P 848b695a07f8402b839455990e0514bc [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:09:18.268649 19646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8649b568deb04ed9aa157a3b4c8a20f1" candidate_uuid: "151984abb40245de8bd371bfa9754712" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cedfa33539b4180a052efef6210d349"
I20260429 14:09:18.268749 19646 raft_consensus.cc:3060] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:09:18.269402 19922 raft_consensus.cc:2468] T 8649b568deb04ed9aa157a3b4c8a20f1 P 848b695a07f8402b839455990e0514bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 151984abb40245de8bd371bfa9754712 in term 1.
I20260429 14:09:18.269407 19646 raft_consensus.cc:2468] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 151984abb40245de8bd371bfa9754712 in term 1.
I20260429 14:09:18.269642 19725 leader_election.cc:304] T 8649b568deb04ed9aa157a3b4c8a20f1 P 151984abb40245de8bd371bfa9754712 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 151984abb40245de8bd371bfa9754712, 1cedfa33539b4180a052efef6210d349; no voters:
I20260429 14:09:18.269742 19983 raft_consensus.cc:2804] T 8649b568deb04ed9aa157a3b4c8a20f1 P 151984abb40245de8bd371bfa9754712 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:09:18.269905 19983 raft_consensus.cc:697] T 8649b568deb04ed9aa157a3b4c8a20f1 P 151984abb40245de8bd371bfa9754712 [term 1 LEADER]: Becoming Leader. State: Replica: 151984abb40245de8bd371bfa9754712, State: Running, Role: LEADER
I20260429 14:09:18.269989 19983 consensus_queue.cc:237] T 8649b568deb04ed9aa157a3b4c8a20f1 P 151984abb40245de8bd371bfa9754712 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848b695a07f8402b839455990e0514bc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35315 } } peers { permanent_uuid: "151984abb40245de8bd371bfa9754712" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38431 } } peers { permanent_uuid: "1cedfa33539b4180a052efef6210d349" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45833 } }
I20260429 14:09:18.270925 19510 catalog_manager.cc:5671] T 8649b568deb04ed9aa157a3b4c8a20f1 P 151984abb40245de8bd371bfa9754712 reported cstate change: term changed from 0 to 1, leader changed from <none> to 151984abb40245de8bd371bfa9754712 (127.2.197.194). New cstate: current_term: 1 leader_uuid: "151984abb40245de8bd371bfa9754712" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848b695a07f8402b839455990e0514bc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "151984abb40245de8bd371bfa9754712" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cedfa33539b4180a052efef6210d349" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45833 } health_report { overall_health: UNKNOWN } } }
I20260429 14:09:18.316094 19976 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:18.313935 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:40616 (local address 127.2.197.194:38431)
0429 14:09:18.314009 (+ 74us) server_negotiation.cc:207] Beginning negotiation
0429 14:09:18.314011 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:09:18.314021 (+ 10us) server_negotiation.cc:408] Connection header received
0429 14:09:18.314086 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:09:18.314086 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:09:18.314100 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:09:18.314150 (+ 50us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:09:18.314545 (+ 395us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:18.315010 (+ 465us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:09:18.315631 (+ 621us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:18.315767 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:18.315795 (+ 28us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:09:18.315889 (+ 94us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:09:18.315956 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0429 14:09:18.316016 (+ 60us) server_negotiation.cc:300] Negotiation successful
0429 14:09:18.316037 (+ 21us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260429 14:09:18.317849 19646 raft_consensus.cc:1275] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [term 1 FOLLOWER]: Refusing update from remote peer 151984abb40245de8bd371bfa9754712: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:09:18.317849 19922 raft_consensus.cc:1275] T 8649b568deb04ed9aa157a3b4c8a20f1 P 848b695a07f8402b839455990e0514bc [term 1 FOLLOWER]: Refusing update from remote peer 151984abb40245de8bd371bfa9754712: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:09:18.318146 19983 consensus_queue.cc:1048] T 8649b568deb04ed9aa157a3b4c8a20f1 P 151984abb40245de8bd371bfa9754712 [LEADER]: Connected to new peer: Peer: permanent_uuid: "848b695a07f8402b839455990e0514bc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:09:18.318320 19986 consensus_queue.cc:1048] T 8649b568deb04ed9aa157a3b4c8a20f1 P 151984abb40245de8bd371bfa9754712 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cedfa33539b4180a052efef6210d349" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:09:18.320214 19988 mvcc.cc:204] Tried to move back new op lower bound from 7280524322066997248 to 7280524321874444288. Current Snapshot: MvccSnapshot[applied={T|T < 7280524322066997248}]
I20260429 14:09:18.322105 19991 mvcc.cc:204] Tried to move back new op lower bound from 7280524322066997248 to 7280524321874444288. Current Snapshot: MvccSnapshot[applied={T|T < 7280524322066997248 or (T in {7280524322066997248})}]
I20260429 14:09:18.322391 19510 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:09:18.322487 19510 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:09:18.323177 19510 catalog_manager.cc:5958] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22: Sending DeleteTablet for 3 replicas of tablet 8649b568deb04ed9aa157a3b4c8a20f1
I20260429 14:09:18.323518 19632 tablet_service.cc:1558] Processing DeleteTablet for tablet 8649b568deb04ed9aa157a3b4c8a20f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:09:18 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:55268
I20260429 14:09:18.323531 19902 tablet_service.cc:1558] Processing DeleteTablet for tablet 8649b568deb04ed9aa157a3b4c8a20f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:09:18 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:48062
I20260429 14:09:18.323709 19752 tablet_service.cc:1558] Processing DeleteTablet for tablet 8649b568deb04ed9aa157a3b4c8a20f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:09:18 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:50918
I20260429 14:09:18.323760 19998 tablet_replica.cc:333] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349: stopping tablet replica
I20260429 14:09:18.323830 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 19548
I20260429 14:09:18.323920 19998 raft_consensus.cc:2243] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:09:18.324026 19998 raft_consensus.cc:2272] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:09:18.324409 20000 tablet_replica.cc:333] T 8649b568deb04ed9aa157a3b4c8a20f1 P 151984abb40245de8bd371bfa9754712: stopping tablet replica
I20260429 14:09:18.324498 20000 raft_consensus.cc:2243] T 8649b568deb04ed9aa157a3b4c8a20f1 P 151984abb40245de8bd371bfa9754712 [term 1 LEADER]: Raft consensus shutting down.
I20260429 14:09:18.324594 20000 raft_consensus.cc:2272] T 8649b568deb04ed9aa157a3b4c8a20f1 P 151984abb40245de8bd371bfa9754712 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:09:18.325006 20000 ts_tablet_manager.cc:1916] T 8649b568deb04ed9aa157a3b4c8a20f1 P 151984abb40245de8bd371bfa9754712: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:09:18.327200 20000 ts_tablet_manager.cc:1929] T 8649b568deb04ed9aa157a3b4c8a20f1 P 151984abb40245de8bd371bfa9754712: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 14:09:18.327278 20000 log.cc:1199] T 8649b568deb04ed9aa157a3b4c8a20f1 P 151984abb40245de8bd371bfa9754712: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/wal/wals/8649b568deb04ed9aa157a3b4c8a20f1
I20260429 14:09:18.327764 20000 ts_tablet_manager.cc:1950] T 8649b568deb04ed9aa157a3b4c8a20f1 P 151984abb40245de8bd371bfa9754712: Deleting consensus metadata
I20260429 14:09:18.328161 19494 catalog_manager.cc:5002] TS 151984abb40245de8bd371bfa9754712 (127.2.197.194:38431): tablet 8649b568deb04ed9aa157a3b4c8a20f1 (table test-table [id=2bbbcbbefb6a41d991e8abf3cd258d50]) successfully deleted
I20260429 14:09:18.330896 19999 tablet_replica.cc:333] T 8649b568deb04ed9aa157a3b4c8a20f1 P 848b695a07f8402b839455990e0514bc: stopping tablet replica
I20260429 14:09:18.331086 19999 raft_consensus.cc:2243] T 8649b568deb04ed9aa157a3b4c8a20f1 P 848b695a07f8402b839455990e0514bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:09:18.331269 19999 raft_consensus.cc:2272] T 8649b568deb04ed9aa157a3b4c8a20f1 P 848b695a07f8402b839455990e0514bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:09:18.331845 19999 ts_tablet_manager.cc:1916] T 8649b568deb04ed9aa157a3b4c8a20f1 P 848b695a07f8402b839455990e0514bc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260429 14:09:18.332461 19496 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.197.193:45833 (error 108)
I20260429 14:09:18.333422 19975 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:18.333301 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:45833 (local address 127.0.0.1:33502)
0429 14:09:18.333354 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:18.333387 (+ 33us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.193:45833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260429 14:09:18.333632 19999 ts_tablet_manager.cc:1929] T 8649b568deb04ed9aa157a3b4c8a20f1 P 848b695a07f8402b839455990e0514bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 14:09:18.333724 19999 log.cc:1199] T 8649b568deb04ed9aa157a3b4c8a20f1 P 848b695a07f8402b839455990e0514bc: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/wal/wals/8649b568deb04ed9aa157a3b4c8a20f1
W20260429 14:09:18.333531 19496 catalog_manager.cc:4729] TS 1cedfa33539b4180a052efef6210d349 (127.2.197.193:45833): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8649b568deb04ed9aa157a3b4c8a20f1: Network error: Client connection negotiation failed: client connection to 127.2.197.193:45833: connect: Connection refused (error 111)
I20260429 14:09:18.334054 19999 ts_tablet_manager.cc:1950] T 8649b568deb04ed9aa157a3b4c8a20f1 P 848b695a07f8402b839455990e0514bc: Deleting consensus metadata
I20260429 14:09:18.334044 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 19703
I20260429 14:09:18.334569 19494 catalog_manager.cc:5002] TS 848b695a07f8402b839455990e0514bc (127.2.197.195:35315): tablet 8649b568deb04ed9aa157a3b4c8a20f1 (table test-table [id=2bbbcbbefb6a41d991e8abf3cd258d50]) successfully deleted
I20260429 14:09:18.339571 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 19839
I20260429 14:09:18.344990 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 19477
I20260429 14:09:38.350847 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:46655
--webserver_interface=127.2.197.254
--webserver_port=32977
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:39857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:46655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:09:38.434110 20002 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:09:38.434391 20002 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:09:38.434422 20002 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:09:38.435971 20002 flags.cc:432] Enabled experimental flag: --authn_token_validity_seconds=20
W20260429 14:09:38.436019 20002 flags.cc:432] Enabled experimental flag: --authz_token_validity_seconds=20
W20260429 14:09:38.436033 20002 flags.cc:432] Enabled experimental flag: --tsk_rotation_seconds=20
W20260429 14:09:38.436043 20002 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:09:38.436056 20002 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:09:38.436069 20002 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:09:38.436127 20002 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:09:38.436144 20002 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:09:38.437968 20002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:39857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:46655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:46655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=32977
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:09:38.438226 20002 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:09:38.438534 20002 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:09:38.441776 20008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:09:38.441813 20002 server_base.cc:1061] running on GCE node
W20260429 14:09:38.441928 20007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:09:38.441826 20010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:09:38.442204 20002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:09:38.442481 20002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:09:38.443645 20002 hybrid_clock.cc:648] HybridClock initialized: now 1777471778443630 us; error 27 us; skew 500 ppm
Apr 29 14:09:38 dist-test-slave-8n77 krb5kdc[18921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471778, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:09:38.446414 20002 init.cc:377] Logged in from keytab as kudu/127.2.197.254@KRBTEST.COM (short username kudu)
I20260429 14:09:38.447003 20002 webserver.cc:492] Webserver started at http://127.2.197.254:32977/ using document root <none> and password file <none>
I20260429 14:09:38.447219 20002 fs_manager.cc:362] Metadata directory not provided
I20260429 14:09:38.447263 20002 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:09:38.448546 20002 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:09:38.449291 20017 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:09:38.449545 20002 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:09:38.449640 20002 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "a17d1d014a204c9b99cb95ef72965f22"
format_stamp: "Formatted at 2026-04-29 14:08:56 on dist-test-slave-8n77"
server_key: "8d0393de5cf257c663414fd2a24b5bda"
server_key_iv: "6ab51389eb13d6c82a55acf23ecdad1a"
server_key_version: "encryptionkey@0"
I20260429 14:09:38.449946 20002 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:09:38.458195 20002 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:09:38.465905 20002 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:09:38.466210 20002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:09:38.470925 20002 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:46655
I20260429 14:09:38.471068 20069 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:46655 every 8 connection(s)
I20260429 14:09:38.471355 20002 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:09:38.472416 20070 sys_catalog.cc:263] Verifying existing consensus state
I20260429 14:09:38.473004 20070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22: Bootstrap starting.
I20260429 14:09:38.476214 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 20002
I20260429 14:09:38.476590 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.193:45833
--local_ip_for_outbound_sockets=127.2.197.193
--tserver_master_addrs=127.2.197.254:46655
--webserver_port=34001
--webserver_interface=127.2.197.193
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:39857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260429 14:09:38.489969 20070 log.cc:826] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22: Log is configured to *not* fsync() on all Append() calls
I20260429 14:09:38.493168 20070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260429 14:09:38.493546 20070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22: Bootstrap complete.
I20260429 14:09:38.496660 20070 raft_consensus.cc:359] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a17d1d014a204c9b99cb95ef72965f22" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 46655 } }
I20260429 14:09:38.496938 20070 raft_consensus.cc:740] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a17d1d014a204c9b99cb95ef72965f22, State: Initialized, Role: FOLLOWER
I20260429 14:09:38.497115 20070 consensus_queue.cc:260] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a17d1d014a204c9b99cb95ef72965f22" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 46655 } }
I20260429 14:09:38.497187 20070 raft_consensus.cc:399] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:09:38.497225 20070 raft_consensus.cc:493] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:09:38.497262 20070 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [term 2 FOLLOWER]: Advancing to term 3
I20260429 14:09:38.498427 20070 raft_consensus.cc:515] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a17d1d014a204c9b99cb95ef72965f22" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 46655 } }
I20260429 14:09:38.498548 20070 leader_election.cc:304] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a17d1d014a204c9b99cb95ef72965f22; no voters:
I20260429 14:09:38.498723 20070 leader_election.cc:290] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260429 14:09:38.498817 20074 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [term 3 FOLLOWER]: Leader election won for term 3
I20260429 14:09:38.498931 20070 sys_catalog.cc:565] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:09:38.499011 20074 raft_consensus.cc:697] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [term 3 LEADER]: Becoming Leader. State: Replica: a17d1d014a204c9b99cb95ef72965f22, State: Running, Role: LEADER
I20260429 14:09:38.499101 20074 consensus_queue.cc:237] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a17d1d014a204c9b99cb95ef72965f22" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 46655 } }
I20260429 14:09:38.499364 20075 sys_catalog.cc:455] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a17d1d014a204c9b99cb95ef72965f22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a17d1d014a204c9b99cb95ef72965f22" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 46655 } } }
I20260429 14:09:38.499471 20075 sys_catalog.cc:458] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [sys.catalog]: This master's current role is: LEADER
I20260429 14:09:38.499826 20079 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:09:38.499367 20076 sys_catalog.cc:455] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a17d1d014a204c9b99cb95ef72965f22. Latest consensus state: current_term: 3 leader_uuid: "a17d1d014a204c9b99cb95ef72965f22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a17d1d014a204c9b99cb95ef72965f22" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 46655 } } }
I20260429 14:09:38.500016 20076 sys_catalog.cc:458] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22 [sys.catalog]: This master's current role is: LEADER
I20260429 14:09:38.500958 20079 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:09:38.501104 20079 catalog_manager.cc:1269] Loaded cluster ID: 97b25306dfc84bfcbe5e5f50aca04bc0
I20260429 14:09:38.501147 20079 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:09:38.502466 20079 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:09:38.502640 20079 catalog_manager.cc:6055] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22: Loaded TSK: 0
I20260429 14:09:38.502686 20079 catalog_manager.cc:6055] T 00000000000000000000000000000000 P a17d1d014a204c9b99cb95ef72965f22: Loaded TSK: 1
F20260429 14:09:38.502866 20079 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=RSAPrivateKey error:04093004:rsa routines:old_rsa_priv_decode:RSA lib:../crypto/rsa/rsa_ameth.c:130 error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f5dbd550dcd google::LogMessage::Fail() at ??:0
@ 0x7f5dbd554b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f5dbd5507cc google::LogMessage::Flush() at ??:0
@ 0x7f5dbd551f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5dbe73e5f6 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f5dbe73bd03 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f5dc0064d9e kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f5dc0064fba kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f5dc0065103 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f5dc0024a37 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f5dc002f833 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f5dbe692400 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f5dbe6883c7 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f5dbe9ee6db start_thread at ??:0
@ 0x7f5dbcaa761f clone at ??:0
W20260429 14:09:38.598315 20072 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:09:38.598575 20072 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:09:38.598614 20072 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:09:38.601030 20072 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:09:38.601104 20072 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:09:38.601148 20072 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.193
I20260429 14:09:38.603909 20072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:39857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.193:45833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.197.193
--webserver_port=34001
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:46655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.193
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:09:38.604177 20072 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:09:38.604393 20072 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:09:38.607622 20111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:09:38.607743 20112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:09:38.607743 20114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:09:38.607911 20072 server_base.cc:1061] running on GCE node
I20260429 14:09:38.608088 20072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:09:38.608330 20072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:09:38.611618 20072 hybrid_clock.cc:648] HybridClock initialized: now 1777471778611584 us; error 42 us; skew 500 ppm
Apr 29 14:09:38 dist-test-slave-8n77 krb5kdc[18921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471778, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:09:38.615073 20072 init.cc:377] Logged in from keytab as kudu/127.2.197.193@KRBTEST.COM (short username kudu)
I20260429 14:09:38.615804 20072 webserver.cc:492] Webserver started at http://127.2.197.193:34001/ using document root <none> and password file <none>
I20260429 14:09:38.616026 20072 fs_manager.cc:362] Metadata directory not provided
I20260429 14:09:38.616086 20072 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:09:38.617712 20072 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:09:38.618484 20121 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:09:38.618774 20072 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:09:38.618860 20072 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/wal
uuid: "1cedfa33539b4180a052efef6210d349"
format_stamp: "Formatted at 2026-04-29 14:08:56 on dist-test-slave-8n77"
server_key: "cfb446128f6c04e58ac71a9d2d12d2a5"
server_key_iv: "7ac92a475fdf7394296d091001b95770"
server_key_version: "encryptionkey@0"
I20260429 14:09:38.619199 20072 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:09:38.652838 20072 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:09:38.655391 20072 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:09:38.655509 20072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:09:38.655807 20072 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:09:38.656255 20128 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260429 14:09:38.657408 20072 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260429 14:09:38.657517 20072 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260429 14:09:38.657572 20072 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260429 14:09:38.658147 20072 ts_tablet_manager.cc:616] Registered 1 tablets
I20260429 14:09:38.658202 20072 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260429 14:09:38.658270 20128 tablet_bootstrap.cc:492] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349: Bootstrap starting.
I20260429 14:09:38.665627 20072 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.193:45833
I20260429 14:09:38.666007 20072 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
I20260429 14:09:38.666404 20235 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.193:45833 every 8 connection(s)
I20260429 14:09:38.670836 20128 log.cc:826] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349: Log is configured to *not* fsync() on all Append() calls
I20260429 14:09:38.673310 20128 tablet_bootstrap.cc:492] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260429 14:09:38.673668 20128 tablet_bootstrap.cc:492] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349: Bootstrap complete.
I20260429 14:09:38.674474 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 20072
I20260429 14:09:38.674778 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.194:38431
--local_ip_for_outbound_sockets=127.2.197.194
--tserver_master_addrs=127.2.197.254:46655
--webserver_port=43419
--webserver_interface=127.2.197.194
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:39857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260429 14:09:38.674867 20128 ts_tablet_manager.cc:1403] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.006s
I20260429 14:09:38.676112 20128 raft_consensus.cc:359] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848b695a07f8402b839455990e0514bc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35315 } } peers { permanent_uuid: "151984abb40245de8bd371bfa9754712" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38431 } } peers { permanent_uuid: "1cedfa33539b4180a052efef6210d349" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45833 } }
I20260429 14:09:38.676420 20128 raft_consensus.cc:740] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cedfa33539b4180a052efef6210d349, State: Initialized, Role: FOLLOWER
I20260429 14:09:38.676542 20128 consensus_queue.cc:260] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848b695a07f8402b839455990e0514bc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35315 } } peers { permanent_uuid: "151984abb40245de8bd371bfa9754712" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38431 } } peers { permanent_uuid: "1cedfa33539b4180a052efef6210d349" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45833 } }
I20260429 14:09:38.676810 20128 ts_tablet_manager.cc:1434] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20260429 14:09:38.772711 20238 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:09:38.666713 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.193:59079)
0429 14:09:38.666869 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:38.666905 (+ 36us) client_negotiation.cc:175] Beginning negotiation
0429 14:09:38.667565 (+ 660us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:09:38.772542 (+104977us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":89,"threads_started":1}
W20260429 14:09:38.772575 20240 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:09:38.772845 20240 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:09:38.772900 20240 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:09:38.772965 20236 heartbeater.cc:646] Failed to heartbeat to 127.2.197.254:46655 (0 consecutive failures): Network error: Failed to ping master at 127.2.197.254:46655: Client connection negotiation failed: client connection to 127.2.197.254:46655: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260429 14:09:38.773381 20238 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:38.773239 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.193:59555)
0429 14:09:38.773320 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:38.773361 (+ 41us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260429 14:09:38.773705 20238 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:38.773617 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.193:38167)
0429 14:09:38.773663 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:38.773685 (+ 22us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260429 14:09:38.773823 20236 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260429 14:09:38.774765 20240 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:09:38.774812 20240 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:09:38.774840 20240 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.194
I20260429 14:09:38.776470 20240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:39857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.194:38431
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.197.194
--webserver_port=43419
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:46655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.194
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:09:38.776656 20240 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:09:38.776785 20240 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:09:38.779268 20249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:09:38.779332 20247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:09:38.779327 20240 server_base.cc:1061] running on GCE node
W20260429 14:09:38.779349 20246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:09:38.779606 20240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:09:38.779827 20240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:09:38.781013 20240 hybrid_clock.cc:648] HybridClock initialized: now 1777471778781000 us; error 32 us; skew 500 ppm
Apr 29 14:09:38 dist-test-slave-8n77 krb5kdc[18921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471778, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:09:38.783495 20240 init.cc:377] Logged in from keytab as kudu/127.2.197.194@KRBTEST.COM (short username kudu)
I20260429 14:09:38.784058 20240 webserver.cc:492] Webserver started at http://127.2.197.194:43419/ using document root <none> and password file <none>
I20260429 14:09:38.784255 20240 fs_manager.cc:362] Metadata directory not provided
I20260429 14:09:38.784329 20240 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:09:38.785542 20240 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:09:38.786157 20256 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:09:38.786592 20240 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:09:38.786695 20240 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/wal
uuid: "151984abb40245de8bd371bfa9754712"
format_stamp: "Formatted at 2026-04-29 14:08:57 on dist-test-slave-8n77"
server_key: "8261b44928bd71816fdb05dee8bf85ca"
server_key_iv: "73ae5cad3f4a5be6c4d3ad4c2df66412"
server_key_version: "encryptionkey@0"
I20260429 14:09:38.786981 20240 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:09:38.808961 20240 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:09:38.811098 20240 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:09:38.811280 20240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:09:38.811530 20240 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:09:38.811846 20240 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:09:38.811901 20240 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:09:38.811944 20240 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:09:38.811993 20240 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:09:38.819003 20240 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.194:38431
I20260429 14:09:38.819060 20369 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.194:38431 every 8 connection(s)
I20260429 14:09:38.819542 20240 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
I20260429 14:09:38.820815 20372 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:38.820453 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.194:34455)
0429 14:09:38.820649 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:38.820720 (+ 71us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":70,"threads_started":1}
W20260429 14:09:38.821094 20370 heartbeater.cc:646] Failed to heartbeat to 127.2.197.254:46655 (0 consecutive failures): Network error: Failed to ping master at 127.2.197.254:46655: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
I20260429 14:09:38.821513 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 20240
I20260429 14:09:38.821734 20372 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:38.821486 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.194:35391)
0429 14:09:38.821621 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:38.821702 (+ 81us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260429 14:09:38.821938 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.195:35315
--local_ip_for_outbound_sockets=127.2.197.195
--tserver_master_addrs=127.2.197.254:46655
--webserver_port=41087
--webserver_interface=127.2.197.195
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:39857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260429 14:09:38.822119 20372 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:38.822024 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.194:55853)
0429 14:09:38.822071 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:38.822095 (+ 24us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260429 14:09:38.822213 20370 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260429 14:09:38.905366 20373 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:09:38.905625 20373 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:09:38.905663 20373 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:09:38.907284 20373 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:09:38.907390 20373 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:09:38.907437 20373 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.195
I20260429 14:09:38.909147 20373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:39857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.195:35315
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.197.195
--webserver_port=41087
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:46655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.195
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:09:38.909442 20373 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:09:38.909643 20373 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:09:38.912343 20378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:09:38.912401 20381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:09:38.912503 20373 server_base.cc:1061] running on GCE node
W20260429 14:09:38.912443 20379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:09:38.912760 20373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:09:38.912955 20373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:09:38.914103 20373 hybrid_clock.cc:648] HybridClock initialized: now 1777471778914083 us; error 38 us; skew 500 ppm
Apr 29 14:09:38 dist-test-slave-8n77 krb5kdc[18921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471778, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:09:38.916751 20373 init.cc:377] Logged in from keytab as kudu/127.2.197.195@KRBTEST.COM (short username kudu)
I20260429 14:09:38.917251 20373 webserver.cc:492] Webserver started at http://127.2.197.195:41087/ using document root <none> and password file <none>
I20260429 14:09:38.917450 20373 fs_manager.cc:362] Metadata directory not provided
I20260429 14:09:38.917492 20373 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:09:38.918793 20373 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260429 14:09:38.919657 20388 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:09:38.919960 20373 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:09:38.920047 20373 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/wal
uuid: "848b695a07f8402b839455990e0514bc"
format_stamp: "Formatted at 2026-04-29 14:08:57 on dist-test-slave-8n77"
server_key: "24e77462343480f84a99ab4a8db83d00"
server_key_iv: "0ac3c7c301aee6eff4466efdef3ad877"
server_key_version: "encryptionkey@0"
I20260429 14:09:38.920316 20373 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:09:38.927950 20373 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:09:38.929848 20373 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:09:38.929955 20373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:09:38.930123 20373 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:09:38.930526 20373 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:09:38.930562 20373 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:09:38.930584 20373 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:09:38.930617 20373 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:09:38.936736 20373 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.195:35315
I20260429 14:09:38.936800 20501 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.195:35315 every 8 connection(s)
I20260429 14:09:38.937109 20373 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
I20260429 14:09:38.938421 20504 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:38.938041 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.195:51033)
0429 14:09:38.938193 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:38.938322 (+ 129us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
W20260429 14:09:38.938671 20502 heartbeater.cc:646] Failed to heartbeat to 127.2.197.254:46655 (0 consecutive failures): Network error: Failed to ping master at 127.2.197.254:46655: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
I20260429 14:09:38.938992 20504 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:38.938893 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.195:54395)
0429 14:09:38.938943 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:38.938971 (+ 28us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260429 14:09:38.939360 20504 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:38.939270 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.195:34975)
0429 14:09:38.939322 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:38.939345 (+ 23us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260429 14:09:38.939666 20502 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260429 14:09:38.947224 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 20373
I20260429 14:09:39.774637 20506 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:39.774357 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.193:53977)
0429 14:09:39.774546 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:39.774590 (+ 44us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":71,"threads_started":1}
I20260429 14:09:39.822831 20507 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:39.822600 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.194:60023)
0429 14:09:39.822741 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:39.822778 (+ 37us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":57,"threads_started":1}
I20260429 14:09:39.940335 20508 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:39.940064 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.195:39227)
0429 14:09:39.940218 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:39.940262 (+ 44us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":62,"threads_started":1}
I20260429 14:09:40.247851 20509 raft_consensus.cc:493] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:09:40.247962 20509 raft_consensus.cc:515] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848b695a07f8402b839455990e0514bc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35315 } } peers { permanent_uuid: "151984abb40245de8bd371bfa9754712" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38431 } } peers { permanent_uuid: "1cedfa33539b4180a052efef6210d349" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45833 } }
I20260429 14:09:40.248369 20509 leader_election.cc:290] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 848b695a07f8402b839455990e0514bc (127.2.197.195:35315), 151984abb40245de8bd371bfa9754712 (127.2.197.194:38431)
Apr 29 14:09:40 dist-test-slave-8n77 krb5kdc[18921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471778, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for kudu/127.2.197.194@KRBTEST.COM
Apr 29 14:09:40 dist-test-slave-8n77 krb5kdc[18921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471778, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for kudu/127.2.197.195@KRBTEST.COM
I20260429 14:09:40.258912 20511 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:40.248744 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:38431 (local address 127.2.197.193:45787)
0429 14:09:40.248846 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:40.248858 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:09:40.248928 (+ 70us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:09:40.249559 (+ 631us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:09:40.249560 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:09:40.249997 (+ 437us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:09:40.250352 (+ 355us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:40.250357 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:40.251585 (+ 1228us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:09:40.251586 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:09:40.251957 (+ 371us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:40.251958 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:40.252097 (+ 139us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:40.252753 (+ 656us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:09:40.252764 (+ 11us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:09:40.254762 (+ 1998us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:09:40.257411 (+ 2649us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:09:40.257417 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:09:40.257419 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:09:40.257634 (+ 215us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:09:40.257892 (+ 258us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:09:40.257892 (+ 0us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:09:40.257893 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:09:40.257982 (+ 89us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:09:40.258422 (+ 440us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:09:40.258427 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:09:40.258563 (+ 136us) client_negotiation.cc:770] Sending connection context
0429 14:09:40.258658 (+ 95us) client_negotiation.cc:241] Negotiation successful
0429 14:09:40.258791 (+ 133us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":51,"threads_started":1}
I20260429 14:09:40.259140 20512 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:40.248839 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:45787 (local address 127.2.197.194:38431)
0429 14:09:40.248953 (+ 114us) server_negotiation.cc:207] Beginning negotiation
0429 14:09:40.248965 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0429 14:09:40.248981 (+ 16us) server_negotiation.cc:408] Connection header received
0429 14:09:40.249392 (+ 411us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:09:40.249395 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:09:40.249425 (+ 30us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:09:40.249547 (+ 122us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:09:40.250469 (+ 922us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:40.251465 (+ 996us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:09:40.252041 (+ 576us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:40.252284 (+ 243us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:40.254910 (+ 2626us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:09:40.254950 (+ 40us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:09:40.254951 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:09:40.254968 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:09:40.257303 (+ 2335us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:09:40.257729 (+ 426us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:09:40.257732 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:09:40.257732 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:09:40.257796 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:09:40.258056 (+ 260us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:09:40.258057 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:09:40.258058 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:09:40.258367 (+ 309us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:09:40.258437 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0429 14:09:40.258735 (+ 298us) server_negotiation.cc:300] Negotiation successful
0429 14:09:40.258873 (+ 138us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":50,"threads_started":1}
I20260429 14:09:40.259254 20510 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:40.248679 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:33323 (local address 127.2.197.195:35315)
0429 14:09:40.248836 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0429 14:09:40.248857 (+ 21us) server_negotiation.cc:400] Waiting for connection header
0429 14:09:40.248877 (+ 20us) server_negotiation.cc:408] Connection header received
0429 14:09:40.249325 (+ 448us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:09:40.249328 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:09:40.249369 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:09:40.249512 (+ 143us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:09:40.250484 (+ 972us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:40.251461 (+ 977us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:09:40.252070 (+ 609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:40.252331 (+ 261us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:40.255495 (+ 3164us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:09:40.255530 (+ 35us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:09:40.255530 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:09:40.255561 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:09:40.257773 (+ 2212us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:09:40.258162 (+ 389us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:09:40.258165 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:09:40.258165 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:09:40.258220 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:09:40.258516 (+ 296us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:09:40.258516 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:09:40.258517 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:09:40.258729 (+ 212us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:09:40.258812 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0429 14:09:40.258994 (+ 182us) server_negotiation.cc:300] Negotiation successful
0429 14:09:40.259125 (+ 131us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260429 14:09:40.259814 20506 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:40.248584 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:35315 (local address 127.2.197.193:33323)
0429 14:09:40.248661 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:40.248676 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 14:09:40.248754 (+ 78us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:09:40.249544 (+ 790us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:09:40.249547 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:09:40.250000 (+ 453us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:09:40.250352 (+ 352us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:40.250357 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:40.251588 (+ 1231us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:09:40.251589 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:09:40.251980 (+ 391us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:40.251981 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:40.252098 (+ 117us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:40.252868 (+ 770us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:09:40.252872 (+ 4us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:09:40.255355 (+ 2483us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:09:40.257893 (+ 2538us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:09:40.257893 (+ 0us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:09:40.257894 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:09:40.258086 (+ 192us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:09:40.258394 (+ 308us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:09:40.258395 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:09:40.258395 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:09:40.258440 (+ 45us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:09:40.258813 (+ 373us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:09:40.258814 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:09:40.258852 (+ 38us) client_negotiation.cc:770] Sending connection context
0429 14:09:40.258924 (+ 72us) client_negotiation.cc:241] Negotiation successful
0429 14:09:40.259690 (+ 766us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
W20260429 14:09:40.259816 20122 leader_election.cc:343] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 151984abb40245de8bd371bfa9754712 (127.2.197.194:38431): Not found: Tablet not found: 8649b568deb04ed9aa157a3b4c8a20f1
W20260429 14:09:40.260433 20122 leader_election.cc:343] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 848b695a07f8402b839455990e0514bc (127.2.197.195:35315): Not found: Tablet not found: 8649b568deb04ed9aa157a3b4c8a20f1
I20260429 14:09:40.260480 20122 leader_election.cc:304] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cedfa33539b4180a052efef6210d349; no voters: 151984abb40245de8bd371bfa9754712, 848b695a07f8402b839455990e0514bc
I20260429 14:09:40.260588 20509 raft_consensus.cc:2749] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260429 14:09:40.775480 20513 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:40.775206 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.193:42587)
0429 14:09:40.775385 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:40.775429 (+ 44us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":66,"threads_started":1}
I20260429 14:09:40.824014 20514 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:40.823658 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.194:38625)
0429 14:09:40.823898 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:40.823951 (+ 53us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":126,"threads_started":1}
I20260429 14:09:40.941196 20515 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:40.940939 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.195:44463)
0429 14:09:40.941101 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:40.941147 (+ 46us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":65,"threads_started":1}
I20260429 14:09:41.776453 20516 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:41.776199 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.193:48373)
0429 14:09:41.776360 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:41.776404 (+ 44us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":75,"threads_started":1}
I20260429 14:09:41.824887 20517 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:41.824598 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.194:42347)
0429 14:09:41.824713 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:41.824757 (+ 44us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":72,"threads_started":1}
I20260429 14:09:41.912192 20518 raft_consensus.cc:493] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:09:41.912365 20518 raft_consensus.cc:515] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848b695a07f8402b839455990e0514bc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35315 } } peers { permanent_uuid: "151984abb40245de8bd371bfa9754712" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38431 } } peers { permanent_uuid: "1cedfa33539b4180a052efef6210d349" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45833 } }
I20260429 14:09:41.912556 20518 leader_election.cc:290] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 848b695a07f8402b839455990e0514bc (127.2.197.195:35315), 151984abb40245de8bd371bfa9754712 (127.2.197.194:38431)
W20260429 14:09:41.913022 20122 leader_election.cc:343] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 848b695a07f8402b839455990e0514bc (127.2.197.195:35315): Not found: Tablet not found: 8649b568deb04ed9aa157a3b4c8a20f1
W20260429 14:09:41.913126 20122 leader_election.cc:343] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 151984abb40245de8bd371bfa9754712 (127.2.197.194:38431): Not found: Tablet not found: 8649b568deb04ed9aa157a3b4c8a20f1
I20260429 14:09:41.913160 20122 leader_election.cc:304] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cedfa33539b4180a052efef6210d349; no voters: 151984abb40245de8bd371bfa9754712, 848b695a07f8402b839455990e0514bc
I20260429 14:09:41.913264 20518 raft_consensus.cc:2749] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260429 14:09:41.942121 20519 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:41.941858 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.195:52911)
0429 14:09:41.942012 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:41.942057 (+ 45us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":67,"threads_started":1}
I20260429 14:09:42.777346 20520 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:42.777096 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.193:32795)
0429 14:09:42.777248 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:42.777291 (+ 43us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":107,"threads_started":1}
I20260429 14:09:42.825783 20521 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:42.825551 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.194:55539)
0429 14:09:42.825684 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:42.825727 (+ 43us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":62,"threads_started":1}
I20260429 14:09:42.943058 20522 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:42.942791 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.195:42187)
0429 14:09:42.942968 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:42.943010 (+ 42us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":127,"threads_started":1}
I20260429 14:09:43.578425 20523 raft_consensus.cc:493] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:09:43.578547 20523 raft_consensus.cc:515] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848b695a07f8402b839455990e0514bc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35315 } } peers { permanent_uuid: "151984abb40245de8bd371bfa9754712" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38431 } } peers { permanent_uuid: "1cedfa33539b4180a052efef6210d349" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45833 } }
I20260429 14:09:43.578751 20523 leader_election.cc:290] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 848b695a07f8402b839455990e0514bc (127.2.197.195:35315), 151984abb40245de8bd371bfa9754712 (127.2.197.194:38431)
W20260429 14:09:43.579239 20122 leader_election.cc:343] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 848b695a07f8402b839455990e0514bc (127.2.197.195:35315): Not found: Tablet not found: 8649b568deb04ed9aa157a3b4c8a20f1
W20260429 14:09:43.579317 20122 leader_election.cc:343] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 151984abb40245de8bd371bfa9754712 (127.2.197.194:38431): Not found: Tablet not found: 8649b568deb04ed9aa157a3b4c8a20f1
I20260429 14:09:43.579347 20122 leader_election.cc:304] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cedfa33539b4180a052efef6210d349; no voters: 151984abb40245de8bd371bfa9754712, 848b695a07f8402b839455990e0514bc
I20260429 14:09:43.579448 20523 raft_consensus.cc:2749] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260429 14:09:43.778245 20524 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:43.777996 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.193:34313)
0429 14:09:43.778154 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:43.778197 (+ 43us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":66,"threads_started":1}
I20260429 14:09:43.826606 20525 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:43.826408 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.194:57305)
0429 14:09:43.826528 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:43.826561 (+ 33us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":59,"threads_started":1}
I20260429 14:09:43.944039 20526 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:43.943757 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.195:40327)
0429 14:09:43.943939 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:43.943993 (+ 54us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":99,"threads_started":1}
I20260429 14:09:44.779114 20527 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:44.778870 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.193:59921)
0429 14:09:44.779016 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:44.779063 (+ 47us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":62,"threads_started":1}
I20260429 14:09:44.827314 20528 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:44.827073 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.194:34393)
0429 14:09:44.827199 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:44.827234 (+ 35us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":61,"threads_started":1}
I20260429 14:09:44.944831 20529 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:44.944595 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.195:45521)
0429 14:09:44.944737 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:44.944777 (+ 40us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":67,"threads_started":1}
I20260429 14:09:45.780073 20530 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:45.779789 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.193:35835)
0429 14:09:45.779980 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:45.780024 (+ 44us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":93,"threads_started":1}
I20260429 14:09:45.828159 20531 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:45.827930 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.194:44983)
0429 14:09:45.828070 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:45.828110 (+ 40us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":73,"threads_started":1}
I20260429 14:09:45.945519 20532 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:45.945307 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.195:45255)
0429 14:09:45.945440 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:45.945473 (+ 33us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
I20260429 14:09:46.780951 20533 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:46.780711 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.193:40795)
0429 14:09:46.780858 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:46.780902 (+ 44us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":63,"threads_started":1}
I20260429 14:09:46.828954 20534 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:46.828710 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.194:42011)
0429 14:09:46.828873 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:46.828910 (+ 37us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":81,"threads_started":1}
I20260429 14:09:46.946262 20535 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:46.946023 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.195:53287)
0429 14:09:46.946182 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:46.946216 (+ 34us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":77,"threads_started":1}
I20260429 14:09:47.498199 20536 raft_consensus.cc:493] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:09:47.498354 20536 raft_consensus.cc:515] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848b695a07f8402b839455990e0514bc" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 35315 } } peers { permanent_uuid: "151984abb40245de8bd371bfa9754712" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38431 } } peers { permanent_uuid: "1cedfa33539b4180a052efef6210d349" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 45833 } }
I20260429 14:09:47.498593 20536 leader_election.cc:290] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 848b695a07f8402b839455990e0514bc (127.2.197.195:35315), 151984abb40245de8bd371bfa9754712 (127.2.197.194:38431)
W20260429 14:09:47.499060 20122 leader_election.cc:343] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 848b695a07f8402b839455990e0514bc (127.2.197.195:35315): Not found: Tablet not found: 8649b568deb04ed9aa157a3b4c8a20f1
W20260429 14:09:47.499142 20122 leader_election.cc:343] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 151984abb40245de8bd371bfa9754712 (127.2.197.194:38431): Not found: Tablet not found: 8649b568deb04ed9aa157a3b4c8a20f1
I20260429 14:09:47.499187 20122 leader_election.cc:304] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cedfa33539b4180a052efef6210d349; no voters: 151984abb40245de8bd371bfa9754712, 848b695a07f8402b839455990e0514bc
I20260429 14:09:47.499277 20536 raft_consensus.cc:2749] T 8649b568deb04ed9aa157a3b4c8a20f1 P 1cedfa33539b4180a052efef6210d349 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260429 14:09:47.781814 20537 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:47.781551 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.193:43475)
0429 14:09:47.781719 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:47.781763 (+ 44us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":71,"threads_started":1}
I20260429 14:09:47.829870 20538 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:47.829584 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.194:53529)
0429 14:09:47.829759 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:47.829803 (+ 44us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":81,"threads_started":1}
I20260429 14:09:47.947047 20539 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:47.946813 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.195:50727)
0429 14:09:47.946962 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:47.946999 (+ 37us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":64,"threads_started":1}
I20260429 14:09:48.782825 20540 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:48.782572 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.193:41995)
0429 14:09:48.782727 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:48.782773 (+ 46us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":65,"threads_started":1}
I20260429 14:09:48.830775 20541 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:48.830533 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.194:51823)
0429 14:09:48.830681 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:48.830726 (+ 45us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":64,"threads_started":1}
I20260429 14:09:48.947962 20542 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:48.947696 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.195:57977)
0429 14:09:48.947863 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:48.947912 (+ 49us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":82,"threads_started":1}
I20260429 14:09:49.783679 20543 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:49.783439 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.193:42409)
0429 14:09:49.783590 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:49.783630 (+ 40us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":60,"threads_started":1}
I20260429 14:09:49.831629 20544 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:49.831379 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.194:49221)
0429 14:09:49.831533 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:49.831581 (+ 48us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":81,"threads_started":1}
I20260429 14:09:49.948887 20545 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:49.948575 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.195:50331)
0429 14:09:49.948792 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:49.948837 (+ 45us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":86,"threads_started":1}
I20260429 14:09:50.784663 20546 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:50.784386 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.193:34381)
0429 14:09:50.784559 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:50.784603 (+ 44us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":80,"threads_started":1}
I20260429 14:09:50.832393 20547 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:50.832158 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.194:54099)
0429 14:09:50.832303 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:50.832346 (+ 43us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":68,"threads_started":1}
I20260429 14:09:50.949683 20548 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:50.949432 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.195:41641)
0429 14:09:50.949591 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:50.949636 (+ 45us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":83,"threads_started":1}
I20260429 14:09:51.785674 20549 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:51.785352 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.193:36291)
0429 14:09:51.785573 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:51.785623 (+ 50us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":88,"threads_started":1}
I20260429 14:09:51.833132 20550 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:51.832913 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.194:44143)
0429 14:09:51.833053 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:51.833085 (+ 32us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
I20260429 14:09:51.950635 20551 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:51.950403 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.195:35651)
0429 14:09:51.950551 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:51.950587 (+ 36us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":87,"threads_started":1}
I20260429 14:09:52.786576 20552 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:52.786215 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.193:57031)
0429 14:09:52.786451 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:52.786527 (+ 76us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":163,"threads_started":1}
I20260429 14:09:52.833967 20553 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:52.833690 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.194:56389)
0429 14:09:52.833853 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:52.833918 (+ 65us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":73,"threads_started":1}
I20260429 14:09:52.951467 20554 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:52.951197 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.195:50787)
0429 14:09:52.951376 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:52.951416 (+ 40us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":120,"threads_started":1}
I20260429 14:09:53.787514 20555 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:53.787288 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.193:44299)
0429 14:09:53.787423 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:53.787466 (+ 43us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
I20260429 14:09:53.834988 20556 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:53.834722 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.194:56499)
0429 14:09:53.834897 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:53.834940 (+ 43us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":71,"threads_started":1}
I20260429 14:09:53.952525 20557 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:53.952295 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:46655 (local address 127.2.197.195:58535)
0429 14:09:53.952432 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:53.952475 (+ 43us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.254:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
I20260429 14:09:53.953114 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 20072
I20260429 14:09:53.958434 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 20240
I20260429 14:09:53.963421 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 20373
2026-04-29T14:09:53Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59590 ms)
[----------] 31 tests from SecurityITest (380474 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:09:54 dist-test-slave-8n77 krb5kdc[20560](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:09:54 dist-test-slave-8n77 krb5kdc[20560](info): set up 2 sockets
Apr 29 14:09:54 dist-test-slave-8n77 krb5kdc[20560](info): commencing operation
krb5kdc: starting...
W20260429 14:09:56.019824 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:09:56 dist-test-slave-8n77 krb5kdc[20560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:09:56Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:09:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:09:56.173939 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:35717
--webserver_interface=127.2.197.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:43173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:35717
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:09:56.250561 20576 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:09:56.250805 20576 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:09:56.250844 20576 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:09:56.252349 20576 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:09:56.252432 20576 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:09:56.252457 20576 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:09:56.252480 20576 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:09:56.252509 20576 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:09:56.254154 20576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:43173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:35717
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:35717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:09:56.254485 20576 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:09:56.254681 20576 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:09:56.257583 20581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:09:56.257784 20576 server_base.cc:1061] running on GCE node
W20260429 14:09:56.257813 20582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:09:56.257843 20584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:09:56.258136 20576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:09:56.258399 20576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:09:56.259557 20576 hybrid_clock.cc:648] HybridClock initialized: now 1777471796259536 us; error 40 us; skew 500 ppm
Apr 29 14:09:56 dist-test-slave-8n77 krb5kdc[20560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471796, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:09:56.262046 20576 init.cc:377] Logged in from keytab as kudu/127.2.197.254@KRBTEST.COM (short username kudu)
I20260429 14:09:56.262643 20576 webserver.cc:492] Webserver started at http://127.2.197.254:44345/ using document root <none> and password file <none>
I20260429 14:09:56.262836 20576 fs_manager.cc:362] Metadata directory not provided
I20260429 14:09:56.262877 20576 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:09:56.262985 20576 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:09:56.263931 20576 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "00a450a851574a89965a4d5b8fd8d2a1"
format_stamp: "Formatted at 2026-04-29 14:09:56 on dist-test-slave-8n77"
server_key: "5b6cfd394724a597bf95c3486343df4d"
server_key_iv: "13ed8239a27a982515158058cdd538cd"
server_key_version: "encryptionkey@0"
I20260429 14:09:56.264251 20576 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "00a450a851574a89965a4d5b8fd8d2a1"
format_stamp: "Formatted at 2026-04-29 14:09:56 on dist-test-slave-8n77"
server_key: "5b6cfd394724a597bf95c3486343df4d"
server_key_iv: "13ed8239a27a982515158058cdd538cd"
server_key_version: "encryptionkey@0"
I20260429 14:09:56.265468 20576 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260429 14:09:56.266168 20591 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:09:56.266530 20576 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260429 14:09:56.266601 20576 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "00a450a851574a89965a4d5b8fd8d2a1"
format_stamp: "Formatted at 2026-04-29 14:09:56 on dist-test-slave-8n77"
server_key: "5b6cfd394724a597bf95c3486343df4d"
server_key_iv: "13ed8239a27a982515158058cdd538cd"
server_key_version: "encryptionkey@0"
I20260429 14:09:56.266744 20576 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:09:56.277760 20576 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:09:56.279765 20576 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:09:56.279897 20576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:09:56.284148 20576 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:35717
I20260429 14:09:56.284233 20643 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:35717 every 8 connection(s)
I20260429 14:09:56.284562 20576 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:09:56.285167 20644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:09:56.287621 20644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00a450a851574a89965a4d5b8fd8d2a1: Bootstrap starting.
I20260429 14:09:56.288198 20644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00a450a851574a89965a4d5b8fd8d2a1: Neither blocks nor log segments found. Creating new log.
I20260429 14:09:56.288461 20644 log.cc:826] T 00000000000000000000000000000000 P 00a450a851574a89965a4d5b8fd8d2a1: Log is configured to *not* fsync() on all Append() calls
I20260429 14:09:56.288664 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 20576
I20260429 14:09:56.288774 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:09:56.288966 2839 external_mini_cluster.cc:1468] Setting key 7146d7136d0e8fbd95bfe9624969f567
I20260429 14:09:56.289237 20644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00a450a851574a89965a4d5b8fd8d2a1: No bootstrap required, opened a new log
I20260429 14:09:56.290736 20644 raft_consensus.cc:359] T 00000000000000000000000000000000 P 00a450a851574a89965a4d5b8fd8d2a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a450a851574a89965a4d5b8fd8d2a1" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 35717 } }
I20260429 14:09:56.290881 20644 raft_consensus.cc:385] T 00000000000000000000000000000000 P 00a450a851574a89965a4d5b8fd8d2a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:09:56.290915 20644 raft_consensus.cc:740] T 00000000000000000000000000000000 P 00a450a851574a89965a4d5b8fd8d2a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a450a851574a89965a4d5b8fd8d2a1, State: Initialized, Role: FOLLOWER
I20260429 14:09:56.291038 20644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00a450a851574a89965a4d5b8fd8d2a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a450a851574a89965a4d5b8fd8d2a1" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 35717 } }
I20260429 14:09:56.291113 20644 raft_consensus.cc:399] T 00000000000000000000000000000000 P 00a450a851574a89965a4d5b8fd8d2a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:09:56.291149 20644 raft_consensus.cc:493] T 00000000000000000000000000000000 P 00a450a851574a89965a4d5b8fd8d2a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:09:56.291198 20644 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 00a450a851574a89965a4d5b8fd8d2a1 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:09:56.291925 20644 raft_consensus.cc:515] T 00000000000000000000000000000000 P 00a450a851574a89965a4d5b8fd8d2a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a450a851574a89965a4d5b8fd8d2a1" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 35717 } }
I20260429 14:09:56.292022 20644 leader_election.cc:304] T 00000000000000000000000000000000 P 00a450a851574a89965a4d5b8fd8d2a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00a450a851574a89965a4d5b8fd8d2a1; no voters:
I20260429 14:09:56.292196 20644 leader_election.cc:290] T 00000000000000000000000000000000 P 00a450a851574a89965a4d5b8fd8d2a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:09:56.292258 20649 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 00a450a851574a89965a4d5b8fd8d2a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:09:56.292430 20649 raft_consensus.cc:697] T 00000000000000000000000000000000 P 00a450a851574a89965a4d5b8fd8d2a1 [term 1 LEADER]: Becoming Leader. State: Replica: 00a450a851574a89965a4d5b8fd8d2a1, State: Running, Role: LEADER
I20260429 14:09:56.292443 20644 sys_catalog.cc:565] T 00000000000000000000000000000000 P 00a450a851574a89965a4d5b8fd8d2a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:09:56.292721 20649 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00a450a851574a89965a4d5b8fd8d2a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a450a851574a89965a4d5b8fd8d2a1" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 35717 } }
I20260429 14:09:56.293648 20650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00a450a851574a89965a4d5b8fd8d2a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00a450a851574a89965a4d5b8fd8d2a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a450a851574a89965a4d5b8fd8d2a1" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 35717 } } }
I20260429 14:09:56.293727 20650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00a450a851574a89965a4d5b8fd8d2a1 [sys.catalog]: This master's current role is: LEADER
I20260429 14:09:56.294021 20660 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:09:56.294143 20651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00a450a851574a89965a4d5b8fd8d2a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00a450a851574a89965a4d5b8fd8d2a1. Latest consensus state: current_term: 1 leader_uuid: "00a450a851574a89965a4d5b8fd8d2a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a450a851574a89965a4d5b8fd8d2a1" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 35717 } } }
I20260429 14:09:56.294363 20651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00a450a851574a89965a4d5b8fd8d2a1 [sys.catalog]: This master's current role is: LEADER
I20260429 14:09:56.294658 20660 catalog_manager.cc:1494] Initializing Kudu cluster ID...
Apr 29 14:09:56 dist-test-slave-8n77 krb5kdc[20560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:09:56.296057 20660 catalog_manager.cc:1357] Generated new cluster ID: 2086353f5bf642ddac7d3fc35c059065
I20260429 14:09:56.296097 20660 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:09:56.299734 20648 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:56.289463 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:45006 (local address 127.2.197.254:35717)
0429 14:09:56.289635 (+ 172us) server_negotiation.cc:207] Beginning negotiation
0429 14:09:56.289643 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0429 14:09:56.290534 (+ 891us) server_negotiation.cc:408] Connection header received
0429 14:09:56.290713 (+ 179us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:09:56.290719 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:09:56.290757 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:09:56.290867 (+ 110us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:09:56.291890 (+ 1023us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:56.293232 (+ 1342us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:09:56.295539 (+ 2307us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:56.295777 (+ 238us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:56.296521 (+ 744us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:09:56.296527 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:09:56.296527 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:09:56.296544 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:09:56.298397 (+ 1853us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:09:56.298759 (+ 362us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:09:56.298762 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:09:56.298762 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:09:56.298807 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:09:56.298973 (+ 166us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:09:56.298973 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:09:56.298974 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:09:56.299147 (+ 173us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:09:56.299294 (+ 147us) server_negotiation.cc:1036] Waiting for connection context
0429 14:09:56.299393 (+ 99us) server_negotiation.cc:300] Negotiation successful
0429 14:09:56.299517 (+ 124us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":71,"threads_started":1}
I20260429 14:09:56.313308 20660 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:09:56.314042 20660 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:09:56.320767 20660 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 00a450a851574a89965a4d5b8fd8d2a1: Generated new TSK 0
I20260429 14:09:56.320952 20660 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:09:56.378361 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.193:0
--local_ip_for_outbound_sockets=127.2.197.193
--webserver_interface=127.2.197.193
--webserver_port=0
--tserver_master_addrs=127.2.197.254:35717
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:43173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:09:56.467324 20672 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:09:56.467547 20672 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:09:56.467592 20672 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:09:56.469310 20672 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:09:56.469408 20672 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:09:56.469455 20672 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.193
I20260429 14:09:56.471408 20672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:43173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.197.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:35717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.193
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:09:56.471699 20672 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:09:56.471897 20672 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:09:56.474573 20677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:09:56.474601 20678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:09:56.474573 20672 server_base.cc:1061] running on GCE node
W20260429 14:09:56.474637 20680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:09:56.475040 20672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:09:56.475286 20672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:09:56.476472 20672 hybrid_clock.cc:648] HybridClock initialized: now 1777471796476445 us; error 41 us; skew 500 ppm
Apr 29 14:09:56 dist-test-slave-8n77 krb5kdc[20560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471796, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:09:56.479372 20672 init.cc:377] Logged in from keytab as kudu/127.2.197.193@KRBTEST.COM (short username kudu)
I20260429 14:09:56.479960 20672 webserver.cc:492] Webserver started at http://127.2.197.193:35631/ using document root <none> and password file <none>
I20260429 14:09:56.480163 20672 fs_manager.cc:362] Metadata directory not provided
I20260429 14:09:56.480228 20672 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:09:56.480336 20672 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:09:56.481276 20672 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-0/data/instance:
uuid: "21d7d2ed7d5b4cff9381c2f47a4b5ac4"
format_stamp: "Formatted at 2026-04-29 14:09:56 on dist-test-slave-8n77"
server_key: "e886f2507c3b0e1d901a2392fa2b3094"
server_key_iv: "dd4f66ef63f02700139313ece23b80be"
server_key_version: "encryptionkey@0"
I20260429 14:09:56.481623 20672 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance:
uuid: "21d7d2ed7d5b4cff9381c2f47a4b5ac4"
format_stamp: "Formatted at 2026-04-29 14:09:56 on dist-test-slave-8n77"
server_key: "e886f2507c3b0e1d901a2392fa2b3094"
server_key_iv: "dd4f66ef63f02700139313ece23b80be"
server_key_version: "encryptionkey@0"
I20260429 14:09:56.482939 20672 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260429 14:09:56.483704 20687 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:09:56.483949 20672 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:09:56.484033 20672 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-0/wal
uuid: "21d7d2ed7d5b4cff9381c2f47a4b5ac4"
format_stamp: "Formatted at 2026-04-29 14:09:56 on dist-test-slave-8n77"
server_key: "e886f2507c3b0e1d901a2392fa2b3094"
server_key_iv: "dd4f66ef63f02700139313ece23b80be"
server_key_version: "encryptionkey@0"
I20260429 14:09:56.484117 20672 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:09:56.500598 20672 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:09:56.502794 20672 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:09:56.502966 20672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:09:56.503196 20672 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:09:56.503566 20672 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:09:56.503623 20672 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:09:56.503669 20672 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:09:56.503701 20672 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:09:56.509636 20672 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.193:35909
I20260429 14:09:56.509713 20800 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.193:35909 every 8 connection(s)
I20260429 14:09:56.510023 20672 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
I20260429 14:09:56.513568 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 20672
I20260429 14:09:56.513713 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance
I20260429 14:09:56.513916 2839 external_mini_cluster.cc:1468] Setting key c2acd87a56112437ba3009b8d0011abe
Apr 29 14:09:56 dist-test-slave-8n77 krb5kdc[20560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471796, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:09:56.519286 20648 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:56.510878 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:37145 (local address 127.2.197.254:35717)
0429 14:09:56.510963 (+ 85us) server_negotiation.cc:207] Beginning negotiation
0429 14:09:56.510965 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:09:56.511118 (+ 153us) server_negotiation.cc:408] Connection header received
0429 14:09:56.511590 (+ 472us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:09:56.511592 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:09:56.511605 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:09:56.511660 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:09:56.512330 (+ 670us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:56.512835 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:09:56.513357 (+ 522us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:56.513583 (+ 226us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:56.516261 (+ 2678us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:09:56.516265 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:09:56.516265 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:09:56.516272 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:09:56.517749 (+ 1477us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:09:56.518265 (+ 516us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:09:56.518266 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:09:56.518267 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:09:56.518347 (+ 80us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:09:56.518607 (+ 260us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:09:56.518608 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:09:56.518608 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:09:56.518708 (+ 100us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:09:56.518761 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
0429 14:09:56.519124 (+ 363us) server_negotiation.cc:300] Negotiation successful
0429 14:09:56.519215 (+ 91us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260429 14:09:56.519519 20803 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:56.510834 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:35717 (local address 127.2.197.193:37145)
0429 14:09:56.511002 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:56.511029 (+ 27us) client_negotiation.cc:175] Beginning negotiation
0429 14:09:56.511502 (+ 473us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:09:56.511696 (+ 194us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:09:56.511699 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:09:56.511998 (+ 299us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:09:56.512250 (+ 252us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:56.512253 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:56.512914 (+ 661us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:09:56.512915 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:09:56.513284 (+ 369us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:56.513285 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:56.513390 (+ 105us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:56.514528 (+ 1138us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:09:56.514541 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:09:56.516159 (+ 1618us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:09:56.517843 (+ 1684us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:09:56.517852 (+ 9us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:09:56.517855 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:09:56.518178 (+ 323us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:09:56.518433 (+ 255us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:09:56.518434 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:09:56.518435 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:09:56.518536 (+ 101us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:09:56.518778 (+ 242us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:09:56.518788 (+ 10us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:09:56.518962 (+ 174us) client_negotiation.cc:770] Sending connection context
0429 14:09:56.519097 (+ 135us) client_negotiation.cc:241] Negotiation successful
0429 14:09:56.519269 (+ 172us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":99,"threads_started":1}
I20260429 14:09:56.519989 20801 heartbeater.cc:344] Connected to a master server at 127.2.197.254:35717
I20260429 14:09:56.520114 20801 heartbeater.cc:461] Registering TS with master...
I20260429 14:09:56.520323 20801 heartbeater.cc:507] Master 127.2.197.254:35717 requested a full tablet report, sending...
I20260429 14:09:56.520908 20608 ts_manager.cc:194] Registered new tserver with Master: 21d7d2ed7d5b4cff9381c2f47a4b5ac4 (127.2.197.193:35909)
I20260429 14:09:56.521912 20608 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.193@KRBTEST.COM'} at 127.2.197.193:37145
WARNING: no policy specified for kudu/127.2.197.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:09:56.570052 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.194:0
--local_ip_for_outbound_sockets=127.2.197.194
--webserver_interface=127.2.197.194
--webserver_port=0
--tserver_master_addrs=127.2.197.254:35717
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:43173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:09:56.650264 20808 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:09:56.650486 20808 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:09:56.650511 20808 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:09:56.652132 20808 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:09:56.652192 20808 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:09:56.652227 20808 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.194
I20260429 14:09:56.653950 20808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:43173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.197.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:35717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.194
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:09:56.654217 20808 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:09:56.654456 20808 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:09:56.657186 20813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:09:56.657641 20814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:09:56.657200 20816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:09:56.657805 20808 server_base.cc:1061] running on GCE node
I20260429 14:09:56.658021 20808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:09:56.658255 20808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:09:56.659443 20808 hybrid_clock.cc:648] HybridClock initialized: now 1777471796659431 us; error 35 us; skew 500 ppm
Apr 29 14:09:56 dist-test-slave-8n77 krb5kdc[20560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471796, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:09:56.662093 20808 init.cc:377] Logged in from keytab as kudu/127.2.197.194@KRBTEST.COM (short username kudu)
I20260429 14:09:56.662775 20808 webserver.cc:492] Webserver started at http://127.2.197.194:46323/ using document root <none> and password file <none>
I20260429 14:09:56.662981 20808 fs_manager.cc:362] Metadata directory not provided
I20260429 14:09:56.663041 20808 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:09:56.663173 20808 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:09:56.664175 20808 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-1/data/instance:
uuid: "90a829bc5a9a4741acc14fb9c74d87cc"
format_stamp: "Formatted at 2026-04-29 14:09:56 on dist-test-slave-8n77"
server_key: "20700988536bacf2597332175ae768fb"
server_key_iv: "846ede26db45b4d2525b133995b79f6c"
server_key_version: "encryptionkey@0"
I20260429 14:09:56.664707 20808 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance:
uuid: "90a829bc5a9a4741acc14fb9c74d87cc"
format_stamp: "Formatted at 2026-04-29 14:09:56 on dist-test-slave-8n77"
server_key: "20700988536bacf2597332175ae768fb"
server_key_iv: "846ede26db45b4d2525b133995b79f6c"
server_key_version: "encryptionkey@0"
I20260429 14:09:56.665988 20808 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260429 14:09:56.666805 20823 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:09:56.667006 20808 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260429 14:09:56.667097 20808 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-1/wal
uuid: "90a829bc5a9a4741acc14fb9c74d87cc"
format_stamp: "Formatted at 2026-04-29 14:09:56 on dist-test-slave-8n77"
server_key: "20700988536bacf2597332175ae768fb"
server_key_iv: "846ede26db45b4d2525b133995b79f6c"
server_key_version: "encryptionkey@0"
I20260429 14:09:56.667177 20808 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:09:56.683336 20808 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:09:56.685387 20808 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:09:56.685597 20808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:09:56.685832 20808 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:09:56.686162 20808 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:09:56.686367 20808 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:09:56.686422 20808 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:09:56.686453 20808 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:09:56.692672 20808 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.194:42309
I20260429 14:09:56.692708 20936 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.194:42309 every 8 connection(s)
I20260429 14:09:56.693033 20808 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
I20260429 14:09:56.695075 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 20808
I20260429 14:09:56.695196 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance
I20260429 14:09:56.695397 2839 external_mini_cluster.cc:1468] Setting key 0a5a23a2794186d87359183d70cd42d1
Apr 29 14:09:56 dist-test-slave-8n77 krb5kdc[20560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471796, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:09:56.703702 20939 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:56.693884 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:35717 (local address 127.2.197.194:48313)
0429 14:09:56.694099 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:56.694138 (+ 39us) client_negotiation.cc:175] Beginning negotiation
0429 14:09:56.694738 (+ 600us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:09:56.695052 (+ 314us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:09:56.695057 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:09:56.695532 (+ 475us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:09:56.695912 (+ 380us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:56.695916 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:56.696692 (+ 776us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:09:56.696693 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:09:56.697316 (+ 623us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:56.697319 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:56.697455 (+ 136us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:56.698575 (+ 1120us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:09:56.698588 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:09:56.700388 (+ 1800us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:09:56.702393 (+ 2005us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:09:56.702399 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:09:56.702401 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:09:56.702628 (+ 227us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:09:56.702856 (+ 228us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:09:56.702857 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:09:56.702858 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:09:56.702927 (+ 69us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:09:56.703208 (+ 281us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:09:56.703213 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:09:56.703337 (+ 124us) client_negotiation.cc:770] Sending connection context
0429 14:09:56.703429 (+ 92us) client_negotiation.cc:241] Negotiation successful
0429 14:09:56.703558 (+ 129us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":115,"threads_started":1}
I20260429 14:09:56.703706 20648 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:56.693894 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:48313 (local address 127.2.197.254:35717)
0429 14:09:56.693973 (+ 79us) server_negotiation.cc:207] Beginning negotiation
0429 14:09:56.693976 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:09:56.694228 (+ 252us) server_negotiation.cc:408] Connection header received
0429 14:09:56.694881 (+ 653us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:09:56.694883 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:09:56.694893 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:09:56.694963 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:09:56.696012 (+ 1049us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:56.696587 (+ 575us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:09:56.697425 (+ 838us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:56.698042 (+ 617us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:56.700489 (+ 2447us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:09:56.700494 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:09:56.700495 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:09:56.700511 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:09:56.702248 (+ 1737us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:09:56.702714 (+ 466us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:09:56.702715 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:09:56.702716 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:09:56.702748 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:09:56.703003 (+ 255us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:09:56.703004 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:09:56.703005 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:09:56.703140 (+ 135us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:09:56.703216 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0429 14:09:56.703481 (+ 265us) server_negotiation.cc:300] Negotiation successful
0429 14:09:56.703594 (+ 113us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
WARNING: no policy specified for kudu/127.2.197.195@KRBTEST.COM; defaulting to no policy
I20260429 14:09:56.704355 20937 heartbeater.cc:344] Connected to a master server at 127.2.197.254:35717
I20260429 14:09:56.704505 20937 heartbeater.cc:461] Registering TS with master...
I20260429 14:09:56.704727 20937 heartbeater.cc:507] Master 127.2.197.254:35717 requested a full tablet report, sending...
I20260429 14:09:56.705269 20608 ts_manager.cc:194] Registered new tserver with Master: 90a829bc5a9a4741acc14fb9c74d87cc (127.2.197.194:42309)
I20260429 14:09:56.705752 20608 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.194@KRBTEST.COM'} at 127.2.197.194:48313
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:09:56.749567 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.195:0
--local_ip_for_outbound_sockets=127.2.197.195
--webserver_interface=127.2.197.195
--webserver_port=0
--tserver_master_addrs=127.2.197.254:35717
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:43173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:09:56.829149 20944 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:09:56.829336 20944 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:09:56.829360 20944 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:09:56.830921 20944 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:09:56.830966 20944 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:09:56.830993 20944 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.195
I20260429 14:09:56.832607 20944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:43173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.197.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:35717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.195
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:09:56.832839 20944 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:09:56.833029 20944 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:09:56.835918 20952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:09:56.835929 20950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:09:56.835928 20949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:09:56.836088 20944 server_base.cc:1061] running on GCE node
I20260429 14:09:56.836376 20944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:09:56.836647 20944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:09:56.837849 20944 hybrid_clock.cc:648] HybridClock initialized: now 1777471796837835 us; error 30 us; skew 500 ppm
Apr 29 14:09:56 dist-test-slave-8n77 krb5kdc[20560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471796, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:09:56.840421 20944 init.cc:377] Logged in from keytab as kudu/127.2.197.195@KRBTEST.COM (short username kudu)
I20260429 14:09:56.840967 20944 webserver.cc:492] Webserver started at http://127.2.197.195:38731/ using document root <none> and password file <none>
I20260429 14:09:56.841149 20944 fs_manager.cc:362] Metadata directory not provided
I20260429 14:09:56.841215 20944 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:09:56.841298 20944 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:09:56.842227 20944 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-2/data/instance:
uuid: "9b895bd9faeb41f99df00ab91689b6c2"
format_stamp: "Formatted at 2026-04-29 14:09:56 on dist-test-slave-8n77"
server_key: "b308adc730ee845a9a6a1aefe2b1b54d"
server_key_iv: "38f5f42d38595dddababf2cb3281cffd"
server_key_version: "encryptionkey@0"
I20260429 14:09:56.842617 20944 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance:
uuid: "9b895bd9faeb41f99df00ab91689b6c2"
format_stamp: "Formatted at 2026-04-29 14:09:56 on dist-test-slave-8n77"
server_key: "b308adc730ee845a9a6a1aefe2b1b54d"
server_key_iv: "38f5f42d38595dddababf2cb3281cffd"
server_key_version: "encryptionkey@0"
I20260429 14:09:56.843842 20944 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:09:56.844620 20959 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:09:56.844980 20944 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:09:56.845049 20944 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-2/wal
uuid: "9b895bd9faeb41f99df00ab91689b6c2"
format_stamp: "Formatted at 2026-04-29 14:09:56 on dist-test-slave-8n77"
server_key: "b308adc730ee845a9a6a1aefe2b1b54d"
server_key_iv: "38f5f42d38595dddababf2cb3281cffd"
server_key_version: "encryptionkey@0"
I20260429 14:09:56.845144 20944 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:09:56.864460 20944 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:09:56.866600 20944 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:09:56.866751 20944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:09:56.866993 20944 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:09:56.867352 20944 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:09:56.867497 20944 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:09:56.867599 20944 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:09:56.867626 20944 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:09:56.874509 20944 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.195:34651
I20260429 14:09:56.874572 21072 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.195:34651 every 8 connection(s)
I20260429 14:09:56.874893 20944 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
Apr 29 14:09:56 dist-test-slave-8n77 krb5kdc[20560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471796, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:09:56.884140 20648 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:56.875659 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:33071 (local address 127.2.197.254:35717)
0429 14:09:56.875729 (+ 70us) server_negotiation.cc:207] Beginning negotiation
0429 14:09:56.875731 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:09:56.875908 (+ 177us) server_negotiation.cc:408] Connection header received
0429 14:09:56.876390 (+ 482us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:09:56.876391 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:09:56.876407 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:09:56.876475 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:09:56.877247 (+ 772us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:56.877911 (+ 664us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:09:56.878732 (+ 821us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:56.879016 (+ 284us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:56.881343 (+ 2327us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:09:56.881349 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:09:56.881349 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:09:56.881357 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:09:56.882809 (+ 1452us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:09:56.883247 (+ 438us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:09:56.883248 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:09:56.883249 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:09:56.883283 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:09:56.883518 (+ 235us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:09:56.883519 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:09:56.883519 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:09:56.883623 (+ 104us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:09:56.883693 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0429 14:09:56.883985 (+ 292us) server_negotiation.cc:300] Negotiation successful
0429 14:09:56.884069 (+ 84us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260429 14:09:56.884265 21075 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:56.875610 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:35717 (local address 127.2.197.195:33071)
0429 14:09:56.875760 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:56.875795 (+ 35us) client_negotiation.cc:175] Beginning negotiation
0429 14:09:56.876267 (+ 472us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:09:56.876561 (+ 294us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:09:56.876565 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:09:56.876866 (+ 301us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:09:56.877124 (+ 258us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:56.877127 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:56.878053 (+ 926us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:09:56.878054 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:09:56.878590 (+ 536us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:56.878593 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:56.878697 (+ 104us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:56.879664 (+ 967us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:09:56.879673 (+ 9us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:09:56.881226 (+ 1553us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:09:56.882906 (+ 1680us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:09:56.882912 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:09:56.882914 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:09:56.883138 (+ 224us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:09:56.883357 (+ 219us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:09:56.883357 (+ 0us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:09:56.883357 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:09:56.883414 (+ 57us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:09:56.883713 (+ 299us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:09:56.883718 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:09:56.883840 (+ 122us) client_negotiation.cc:770] Sending connection context
0429 14:09:56.883942 (+ 102us) client_negotiation.cc:241] Negotiation successful
0429 14:09:56.884059 (+ 117us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260429 14:09:56.884876 21073 heartbeater.cc:344] Connected to a master server at 127.2.197.254:35717
I20260429 14:09:56.884943 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 20944
I20260429 14:09:56.884999 21073 heartbeater.cc:461] Registering TS with master...
I20260429 14:09:56.885038 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance
I20260429 14:09:56.885212 2839 external_mini_cluster.cc:1468] Setting key 992287ed1ac4ae70b04030c5c89b9f67
I20260429 14:09:56.885236 21073 heartbeater.cc:507] Master 127.2.197.254:35717 requested a full tablet report, sending...
I20260429 14:09:56.885947 20607 ts_manager.cc:194] Registered new tserver with Master: 9b895bd9faeb41f99df00ab91689b6c2 (127.2.197.195:34651)
I20260429 14:09:56.886847 20607 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.195@KRBTEST.COM'} at 127.2.197.195:33071
I20260429 14:09:56.886904 2839 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260429 14:09:56.894619 20648 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:56.888465 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:45012 (local address 127.2.197.254:35717)
0429 14:09:56.888537 (+ 72us) server_negotiation.cc:207] Beginning negotiation
0429 14:09:56.888539 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:09:56.888689 (+ 150us) server_negotiation.cc:408] Connection header received
0429 14:09:56.888715 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:09:56.888716 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:09:56.888725 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:09:56.888776 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:09:56.889470 (+ 694us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:56.889967 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:09:56.890657 (+ 690us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:56.890793 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:56.891824 (+ 1031us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:09:56.891827 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:09:56.891828 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:09:56.891834 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:09:56.893268 (+ 1434us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:09:56.893679 (+ 411us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:09:56.893680 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:09:56.893680 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:09:56.893703 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:09:56.893894 (+ 191us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:09:56.893895 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:09:56.893895 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:09:56.894006 (+ 111us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:09:56.894088 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0429 14:09:56.894418 (+ 330us) server_negotiation.cc:300] Negotiation successful
0429 14:09:56.894513 (+ 95us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260429 14:09:56.896541 20607 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45012:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 14:09:56.897920 20607 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 14:09:56.904049 21085 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:56.900495 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:34651 (local address 127.0.0.1:49326)
0429 14:09:56.900787 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:56.900821 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0429 14:09:56.900905 (+ 84us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:09:56.901377 (+ 472us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:09:56.901378 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:09:56.901381 (+ 3us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:09:56.901631 (+ 250us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:56.901633 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:56.902993 (+ 1360us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:09:56.902994 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:09:56.903731 (+ 737us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:56.903733 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:56.903822 (+ 89us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:56.903862 (+ 40us) client_negotiation.cc:770] Sending connection context
0429 14:09:56.903910 (+ 48us) client_negotiation.cc:241] Negotiation successful
0429 14:09:56.903956 (+ 46us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":77,"threads_started":1}
I20260429 14:09:56.904454 21087 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:56.900714 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49326 (local address 127.2.197.195:34651)
0429 14:09:56.900899 (+ 185us) server_negotiation.cc:207] Beginning negotiation
0429 14:09:56.900908 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0429 14:09:56.900922 (+ 14us) server_negotiation.cc:408] Connection header received
0429 14:09:56.901191 (+ 269us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:09:56.901194 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:09:56.901211 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:09:56.901258 (+ 47us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:09:56.901759 (+ 501us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:56.902908 (+ 1149us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:09:56.903841 (+ 933us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:56.904302 (+ 461us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:56.904340 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0429 14:09:56.904373 (+ 33us) server_negotiation.cc:300] Negotiation successful
0429 14:09:56.904399 (+ 26us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":81,"threads_started":1}
I20260429 14:09:56.905103 21086 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:56.900694 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36134 (local address 127.2.197.194:42309)
0429 14:09:56.901009 (+ 315us) server_negotiation.cc:207] Beginning negotiation
0429 14:09:56.901015 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 14:09:56.901022 (+ 7us) server_negotiation.cc:408] Connection header received
0429 14:09:56.901052 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:09:56.901053 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:09:56.901065 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:09:56.901157 (+ 92us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:09:56.901570 (+ 413us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:56.902786 (+ 1216us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:09:56.904037 (+ 1251us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:56.904910 (+ 873us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:56.904946 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0429 14:09:56.904976 (+ 30us) server_negotiation.cc:300] Negotiation successful
0429 14:09:56.905018 (+ 42us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":86,"threads_started":1}
I20260429 14:09:56.905037 21007 tablet_service.cc:1511] Processing CreateTablet for tablet e41b5b5b3a284cf19e626ff63ff24289 (DEFAULT_TABLE table=test-table [id=3b64b8ea533d41e78baf725d74fa787d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:09:56.905259 21084 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:56.900552 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:42309 (local address 127.0.0.1:36134)
0429 14:09:56.900853 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:56.900862 (+ 9us) client_negotiation.cc:175] Beginning negotiation
0429 14:09:56.900910 (+ 48us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:09:56.901169 (+ 259us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:09:56.901172 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:09:56.901174 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:09:56.901481 (+ 307us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:56.901484 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:56.902867 (+ 1383us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:09:56.902868 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:09:56.903750 (+ 882us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:56.903752 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:56.903825 (+ 73us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:56.903863 (+ 38us) client_negotiation.cc:770] Sending connection context
0429 14:09:56.903912 (+ 49us) client_negotiation.cc:241] Negotiation successful
0429 14:09:56.903961 (+ 49us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":141,"threads_started":1}
I20260429 14:09:56.905332 21007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e41b5b5b3a284cf19e626ff63ff24289. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:09:56.905598 21088 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:56.900806 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:35909 (local address 127.0.0.1:43962)
0429 14:09:56.901349 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:56.901359 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 14:09:56.901416 (+ 57us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:09:56.901611 (+ 195us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:09:56.901612 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:09:56.901617 (+ 5us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:09:56.901808 (+ 191us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:56.901809 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:56.902770 (+ 961us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:09:56.902771 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:09:56.903756 (+ 985us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:56.903757 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:56.903828 (+ 71us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:56.903863 (+ 35us) client_negotiation.cc:770] Sending connection context
0429 14:09:56.903912 (+ 49us) client_negotiation.cc:241] Negotiation successful
0429 14:09:56.903961 (+ 49us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":62,"threads_started":1}
I20260429 14:09:56.905867 20871 tablet_service.cc:1511] Processing CreateTablet for tablet e41b5b5b3a284cf19e626ff63ff24289 (DEFAULT_TABLE table=test-table [id=3b64b8ea533d41e78baf725d74fa787d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:09:56.906090 20871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e41b5b5b3a284cf19e626ff63ff24289. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:09:56.907723 21090 tablet_bootstrap.cc:492] T e41b5b5b3a284cf19e626ff63ff24289 P 9b895bd9faeb41f99df00ab91689b6c2: Bootstrap starting.
I20260429 14:09:56.907852 21089 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:56.901016 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:43962 (local address 127.2.197.193:35909)
0429 14:09:56.901195 (+ 179us) server_negotiation.cc:207] Beginning negotiation
0429 14:09:56.901203 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0429 14:09:56.901471 (+ 268us) server_negotiation.cc:408] Connection header received
0429 14:09:56.901514 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:09:56.901517 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:09:56.901533 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:09:56.901596 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:09:56.901889 (+ 293us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:56.902684 (+ 795us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:09:56.904028 (+ 1344us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:56.904536 (+ 508us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:56.904604 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
0429 14:09:56.904636 (+ 32us) server_negotiation.cc:300] Negotiation successful
0429 14:09:56.907774 (+ 3138us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":106,"threads_started":1}
I20260429 14:09:56.908316 21091 tablet_bootstrap.cc:492] T e41b5b5b3a284cf19e626ff63ff24289 P 90a829bc5a9a4741acc14fb9c74d87cc: Bootstrap starting.
I20260429 14:09:56.908422 20735 tablet_service.cc:1511] Processing CreateTablet for tablet e41b5b5b3a284cf19e626ff63ff24289 (DEFAULT_TABLE table=test-table [id=3b64b8ea533d41e78baf725d74fa787d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:09:56.908567 21090 tablet_bootstrap.cc:654] T e41b5b5b3a284cf19e626ff63ff24289 P 9b895bd9faeb41f99df00ab91689b6c2: Neither blocks nor log segments found. Creating new log.
I20260429 14:09:56.908635 20735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e41b5b5b3a284cf19e626ff63ff24289. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:09:56.908890 21090 log.cc:826] T e41b5b5b3a284cf19e626ff63ff24289 P 9b895bd9faeb41f99df00ab91689b6c2: Log is configured to *not* fsync() on all Append() calls
I20260429 14:09:56.908951 21091 tablet_bootstrap.cc:654] T e41b5b5b3a284cf19e626ff63ff24289 P 90a829bc5a9a4741acc14fb9c74d87cc: Neither blocks nor log segments found. Creating new log.
I20260429 14:09:56.909215 21091 log.cc:826] T e41b5b5b3a284cf19e626ff63ff24289 P 90a829bc5a9a4741acc14fb9c74d87cc: Log is configured to *not* fsync() on all Append() calls
I20260429 14:09:56.909670 21090 tablet_bootstrap.cc:492] T e41b5b5b3a284cf19e626ff63ff24289 P 9b895bd9faeb41f99df00ab91689b6c2: No bootstrap required, opened a new log
I20260429 14:09:56.909718 21090 ts_tablet_manager.cc:1403] T e41b5b5b3a284cf19e626ff63ff24289 P 9b895bd9faeb41f99df00ab91689b6c2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260429 14:09:56.909894 21091 tablet_bootstrap.cc:492] T e41b5b5b3a284cf19e626ff63ff24289 P 90a829bc5a9a4741acc14fb9c74d87cc: No bootstrap required, opened a new log
I20260429 14:09:56.909955 21091 ts_tablet_manager.cc:1403] T e41b5b5b3a284cf19e626ff63ff24289 P 90a829bc5a9a4741acc14fb9c74d87cc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260429 14:09:56.911087 21090 raft_consensus.cc:359] T e41b5b5b3a284cf19e626ff63ff24289 P 9b895bd9faeb41f99df00ab91689b6c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b895bd9faeb41f99df00ab91689b6c2" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 34651 } } peers { permanent_uuid: "90a829bc5a9a4741acc14fb9c74d87cc" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 42309 } } peers { permanent_uuid: "21d7d2ed7d5b4cff9381c2f47a4b5ac4" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 35909 } }
I20260429 14:09:56.911242 21090 raft_consensus.cc:385] T e41b5b5b3a284cf19e626ff63ff24289 P 9b895bd9faeb41f99df00ab91689b6c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:09:56.911281 21090 raft_consensus.cc:740] T e41b5b5b3a284cf19e626ff63ff24289 P 9b895bd9faeb41f99df00ab91689b6c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b895bd9faeb41f99df00ab91689b6c2, State: Initialized, Role: FOLLOWER
I20260429 14:09:56.911382 21090 consensus_queue.cc:260] T e41b5b5b3a284cf19e626ff63ff24289 P 9b895bd9faeb41f99df00ab91689b6c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b895bd9faeb41f99df00ab91689b6c2" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 34651 } } peers { permanent_uuid: "90a829bc5a9a4741acc14fb9c74d87cc" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 42309 } } peers { permanent_uuid: "21d7d2ed7d5b4cff9381c2f47a4b5ac4" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 35909 } }
I20260429 14:09:56.911613 21090 ts_tablet_manager.cc:1434] T e41b5b5b3a284cf19e626ff63ff24289 P 9b895bd9faeb41f99df00ab91689b6c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260429 14:09:56.911691 21094 tablet_bootstrap.cc:492] T e41b5b5b3a284cf19e626ff63ff24289 P 21d7d2ed7d5b4cff9381c2f47a4b5ac4: Bootstrap starting.
I20260429 14:09:56.911756 21073 heartbeater.cc:499] Master 127.2.197.254:35717 was elected leader, sending a full tablet report...
I20260429 14:09:56.911994 21091 raft_consensus.cc:359] T e41b5b5b3a284cf19e626ff63ff24289 P 90a829bc5a9a4741acc14fb9c74d87cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b895bd9faeb41f99df00ab91689b6c2" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 34651 } } peers { permanent_uuid: "90a829bc5a9a4741acc14fb9c74d87cc" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 42309 } } peers { permanent_uuid: "21d7d2ed7d5b4cff9381c2f47a4b5ac4" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 35909 } }
I20260429 14:09:56.912158 21091 raft_consensus.cc:385] T e41b5b5b3a284cf19e626ff63ff24289 P 90a829bc5a9a4741acc14fb9c74d87cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:09:56.912199 21091 raft_consensus.cc:740] T e41b5b5b3a284cf19e626ff63ff24289 P 90a829bc5a9a4741acc14fb9c74d87cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a829bc5a9a4741acc14fb9c74d87cc, State: Initialized, Role: FOLLOWER
I20260429 14:09:56.912278 21091 consensus_queue.cc:260] T e41b5b5b3a284cf19e626ff63ff24289 P 90a829bc5a9a4741acc14fb9c74d87cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b895bd9faeb41f99df00ab91689b6c2" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 34651 } } peers { permanent_uuid: "90a829bc5a9a4741acc14fb9c74d87cc" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 42309 } } peers { permanent_uuid: "21d7d2ed7d5b4cff9381c2f47a4b5ac4" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 35909 } }
I20260429 14:09:56.912421 21094 tablet_bootstrap.cc:654] T e41b5b5b3a284cf19e626ff63ff24289 P 21d7d2ed7d5b4cff9381c2f47a4b5ac4: Neither blocks nor log segments found. Creating new log.
I20260429 14:09:56.912509 21091 ts_tablet_manager.cc:1434] T e41b5b5b3a284cf19e626ff63ff24289 P 90a829bc5a9a4741acc14fb9c74d87cc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260429 14:09:56.912678 20937 heartbeater.cc:499] Master 127.2.197.254:35717 was elected leader, sending a full tablet report...
I20260429 14:09:56.912766 21094 log.cc:826] T e41b5b5b3a284cf19e626ff63ff24289 P 21d7d2ed7d5b4cff9381c2f47a4b5ac4: Log is configured to *not* fsync() on all Append() calls
I20260429 14:09:56.913805 21094 tablet_bootstrap.cc:492] T e41b5b5b3a284cf19e626ff63ff24289 P 21d7d2ed7d5b4cff9381c2f47a4b5ac4: No bootstrap required, opened a new log
I20260429 14:09:56.913888 21094 ts_tablet_manager.cc:1403] T e41b5b5b3a284cf19e626ff63ff24289 P 21d7d2ed7d5b4cff9381c2f47a4b5ac4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260429 14:09:56.915205 21094 raft_consensus.cc:359] T e41b5b5b3a284cf19e626ff63ff24289 P 21d7d2ed7d5b4cff9381c2f47a4b5ac4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b895bd9faeb41f99df00ab91689b6c2" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 34651 } } peers { permanent_uuid: "90a829bc5a9a4741acc14fb9c74d87cc" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 42309 } } peers { permanent_uuid: "21d7d2ed7d5b4cff9381c2f47a4b5ac4" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 35909 } }
I20260429 14:09:56.915329 21094 raft_consensus.cc:385] T e41b5b5b3a284cf19e626ff63ff24289 P 21d7d2ed7d5b4cff9381c2f47a4b5ac4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:09:56.915349 21094 raft_consensus.cc:740] T e41b5b5b3a284cf19e626ff63ff24289 P 21d7d2ed7d5b4cff9381c2f47a4b5ac4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21d7d2ed7d5b4cff9381c2f47a4b5ac4, State: Initialized, Role: FOLLOWER
I20260429 14:09:56.915467 21094 consensus_queue.cc:260] T e41b5b5b3a284cf19e626ff63ff24289 P 21d7d2ed7d5b4cff9381c2f47a4b5ac4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b895bd9faeb41f99df00ab91689b6c2" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 34651 } } peers { permanent_uuid: "90a829bc5a9a4741acc14fb9c74d87cc" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 42309 } } peers { permanent_uuid: "21d7d2ed7d5b4cff9381c2f47a4b5ac4" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 35909 } }
I20260429 14:09:56.915740 21094 ts_tablet_manager.cc:1434] T e41b5b5b3a284cf19e626ff63ff24289 P 21d7d2ed7d5b4cff9381c2f47a4b5ac4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260429 14:09:56.915804 20801 heartbeater.cc:499] Master 127.2.197.254:35717 was elected leader, sending a full tablet report...
W20260429 14:09:56.943639 20938 tablet.cc:2404] T e41b5b5b3a284cf19e626ff63ff24289 P 90a829bc5a9a4741acc14fb9c74d87cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260429 14:09:57.010828 20802 tablet.cc:2404] T e41b5b5b3a284cf19e626ff63ff24289 P 21d7d2ed7d5b4cff9381c2f47a4b5ac4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 14:09:57.051728 21096 raft_consensus.cc:493] T e41b5b5b3a284cf19e626ff63ff24289 P 90a829bc5a9a4741acc14fb9c74d87cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:09:57.051846 21096 raft_consensus.cc:515] T e41b5b5b3a284cf19e626ff63ff24289 P 90a829bc5a9a4741acc14fb9c74d87cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b895bd9faeb41f99df00ab91689b6c2" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 34651 } } peers { permanent_uuid: "90a829bc5a9a4741acc14fb9c74d87cc" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 42309 } } peers { permanent_uuid: "21d7d2ed7d5b4cff9381c2f47a4b5ac4" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 35909 } }
I20260429 14:09:57.052276 21096 leader_election.cc:290] T e41b5b5b3a284cf19e626ff63ff24289 P 90a829bc5a9a4741acc14fb9c74d87cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b895bd9faeb41f99df00ab91689b6c2 (127.2.197.195:34651), 21d7d2ed7d5b4cff9381c2f47a4b5ac4 (127.2.197.193:35909)
I20260429 14:09:57.054612 20939 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:57.052517 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:34651 (local address 127.2.197.194:35929)
0429 14:09:57.052576 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:57.052589 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 14:09:57.052636 (+ 47us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:09:57.052817 (+ 181us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:09:57.052818 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:09:57.052820 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:09:57.053007 (+ 187us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:57.053009 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.053722 (+ 713us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:09:57.053723 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:09:57.054359 (+ 636us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:57.054362 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.054462 (+ 100us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:57.054466 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 14:09:57.054501 (+ 35us) client_negotiation.cc:241] Negotiation successful
0429 14:09:57.054541 (+ 40us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260429 14:09:57.055027 21087 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:57.052623 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:35929 (local address 127.2.197.195:34651)
0429 14:09:57.052684 (+ 61us) server_negotiation.cc:207] Beginning negotiation
0429 14:09:57.052686 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:09:57.052694 (+ 8us) server_negotiation.cc:408] Connection header received
0429 14:09:57.052722 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:09:57.052723 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:09:57.052733 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:09:57.052810 (+ 77us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:09:57.053113 (+ 303us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.053623 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:09:57.054447 (+ 824us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.054931 (+ 484us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:57.054938 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
0429 14:09:57.054959 (+ 21us) server_negotiation.cc:300] Negotiation successful
0429 14:09:57.054982 (+ 23us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260429 14:09:57.055344 21027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e41b5b5b3a284cf19e626ff63ff24289" candidate_uuid: "90a829bc5a9a4741acc14fb9c74d87cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b895bd9faeb41f99df00ab91689b6c2" is_pre_election: true
I20260429 14:09:57.055474 21099 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:57.052528 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:35909 (local address 127.2.197.194:48897)
0429 14:09:57.052732 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:57.052741 (+ 9us) client_negotiation.cc:175] Beginning negotiation
0429 14:09:57.052843 (+ 102us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:09:57.053003 (+ 160us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:09:57.053003 (+ 0us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:09:57.053005 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:09:57.053237 (+ 232us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:57.053239 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.053937 (+ 698us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:09:57.053938 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:09:57.054647 (+ 709us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:57.054649 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.055278 (+ 629us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:57.055282 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 14:09:57.055357 (+ 75us) client_negotiation.cc:241] Negotiation successful
0429 14:09:57.055395 (+ 38us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":11,"thread_start_us":111,"threads_started":1}
I20260429 14:09:57.055475 21089 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:57.052615 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:48897 (local address 127.2.197.193:35909)
0429 14:09:57.052702 (+ 87us) server_negotiation.cc:207] Beginning negotiation
0429 14:09:57.052705 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:09:57.052826 (+ 121us) server_negotiation.cc:408] Connection header received
0429 14:09:57.052912 (+ 86us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:09:57.052913 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:09:57.052923 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:09:57.052999 (+ 76us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:09:57.053323 (+ 324us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.053844 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:09:57.054797 (+ 953us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.055238 (+ 441us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:57.055251 (+ 13us) server_negotiation.cc:1036] Waiting for connection context
0429 14:09:57.055365 (+ 114us) server_negotiation.cc:300] Negotiation successful
0429 14:09:57.055407 (+ 42us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260429 14:09:57.055503 21027 raft_consensus.cc:2468] T e41b5b5b3a284cf19e626ff63ff24289 P 9b895bd9faeb41f99df00ab91689b6c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a829bc5a9a4741acc14fb9c74d87cc in term 0.
I20260429 14:09:57.055799 20825 leader_election.cc:304] T e41b5b5b3a284cf19e626ff63ff24289 P 90a829bc5a9a4741acc14fb9c74d87cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90a829bc5a9a4741acc14fb9c74d87cc, 9b895bd9faeb41f99df00ab91689b6c2; no voters:
I20260429 14:09:57.055864 20755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e41b5b5b3a284cf19e626ff63ff24289" candidate_uuid: "90a829bc5a9a4741acc14fb9c74d87cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21d7d2ed7d5b4cff9381c2f47a4b5ac4" is_pre_election: true
I20260429 14:09:57.055964 21096 raft_consensus.cc:2804] T e41b5b5b3a284cf19e626ff63ff24289 P 90a829bc5a9a4741acc14fb9c74d87cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 14:09:57.055981 20755 raft_consensus.cc:2468] T e41b5b5b3a284cf19e626ff63ff24289 P 21d7d2ed7d5b4cff9381c2f47a4b5ac4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a829bc5a9a4741acc14fb9c74d87cc in term 0.
I20260429 14:09:57.056036 21096 raft_consensus.cc:493] T e41b5b5b3a284cf19e626ff63ff24289 P 90a829bc5a9a4741acc14fb9c74d87cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 14:09:57.056069 21096 raft_consensus.cc:3060] T e41b5b5b3a284cf19e626ff63ff24289 P 90a829bc5a9a4741acc14fb9c74d87cc [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:09:57.056850 21096 raft_consensus.cc:515] T e41b5b5b3a284cf19e626ff63ff24289 P 90a829bc5a9a4741acc14fb9c74d87cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b895bd9faeb41f99df00ab91689b6c2" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 34651 } } peers { permanent_uuid: "90a829bc5a9a4741acc14fb9c74d87cc" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 42309 } } peers { permanent_uuid: "21d7d2ed7d5b4cff9381c2f47a4b5ac4" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 35909 } }
I20260429 14:09:57.057030 21096 leader_election.cc:290] T e41b5b5b3a284cf19e626ff63ff24289 P 90a829bc5a9a4741acc14fb9c74d87cc [CANDIDATE]: Term 1 election: Requested vote from peers 9b895bd9faeb41f99df00ab91689b6c2 (127.2.197.195:34651), 21d7d2ed7d5b4cff9381c2f47a4b5ac4 (127.2.197.193:35909)
I20260429 14:09:57.057184 20755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e41b5b5b3a284cf19e626ff63ff24289" candidate_uuid: "90a829bc5a9a4741acc14fb9c74d87cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21d7d2ed7d5b4cff9381c2f47a4b5ac4"
I20260429 14:09:57.057227 21027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e41b5b5b3a284cf19e626ff63ff24289" candidate_uuid: "90a829bc5a9a4741acc14fb9c74d87cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b895bd9faeb41f99df00ab91689b6c2"
I20260429 14:09:57.057293 20755 raft_consensus.cc:3060] T e41b5b5b3a284cf19e626ff63ff24289 P 21d7d2ed7d5b4cff9381c2f47a4b5ac4 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:09:57.057291 21027 raft_consensus.cc:3060] T e41b5b5b3a284cf19e626ff63ff24289 P 9b895bd9faeb41f99df00ab91689b6c2 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:09:57.057977 21027 raft_consensus.cc:2468] T e41b5b5b3a284cf19e626ff63ff24289 P 9b895bd9faeb41f99df00ab91689b6c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a829bc5a9a4741acc14fb9c74d87cc in term 1.
I20260429 14:09:57.058069 20755 raft_consensus.cc:2468] T e41b5b5b3a284cf19e626ff63ff24289 P 21d7d2ed7d5b4cff9381c2f47a4b5ac4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a829bc5a9a4741acc14fb9c74d87cc in term 1.
I20260429 14:09:57.058178 20825 leader_election.cc:304] T e41b5b5b3a284cf19e626ff63ff24289 P 90a829bc5a9a4741acc14fb9c74d87cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90a829bc5a9a4741acc14fb9c74d87cc, 9b895bd9faeb41f99df00ab91689b6c2; no voters:
I20260429 14:09:57.058329 21096 raft_consensus.cc:2804] T e41b5b5b3a284cf19e626ff63ff24289 P 90a829bc5a9a4741acc14fb9c74d87cc [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:09:57.058485 21096 raft_consensus.cc:697] T e41b5b5b3a284cf19e626ff63ff24289 P 90a829bc5a9a4741acc14fb9c74d87cc [term 1 LEADER]: Becoming Leader. State: Replica: 90a829bc5a9a4741acc14fb9c74d87cc, State: Running, Role: LEADER
I20260429 14:09:57.058601 21096 consensus_queue.cc:237] T e41b5b5b3a284cf19e626ff63ff24289 P 90a829bc5a9a4741acc14fb9c74d87cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b895bd9faeb41f99df00ab91689b6c2" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 34651 } } peers { permanent_uuid: "90a829bc5a9a4741acc14fb9c74d87cc" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 42309 } } peers { permanent_uuid: "21d7d2ed7d5b4cff9381c2f47a4b5ac4" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 35909 } }
I20260429 14:09:57.059396 20607 catalog_manager.cc:5671] T e41b5b5b3a284cf19e626ff63ff24289 P 90a829bc5a9a4741acc14fb9c74d87cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 90a829bc5a9a4741acc14fb9c74d87cc (127.2.197.194). New cstate: current_term: 1 leader_uuid: "90a829bc5a9a4741acc14fb9c74d87cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b895bd9faeb41f99df00ab91689b6c2" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 34651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a829bc5a9a4741acc14fb9c74d87cc" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 42309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d7d2ed7d5b4cff9381c2f47a4b5ac4" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 35909 } health_report { overall_health: UNKNOWN } } }
I20260429 14:09:57.090152 21086 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:57.087866 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36146 (local address 127.2.197.194:42309)
0429 14:09:57.087936 (+ 70us) server_negotiation.cc:207] Beginning negotiation
0429 14:09:57.087938 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:09:57.087947 (+ 9us) server_negotiation.cc:408] Connection header received
0429 14:09:57.087995 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:09:57.087998 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:09:57.088017 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:09:57.088114 (+ 97us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:09:57.088401 (+ 287us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.088873 (+ 472us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:09:57.089594 (+ 721us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.089755 (+ 161us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:57.089787 (+ 32us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:09:57.089880 (+ 93us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:09:57.089973 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0429 14:09:57.090036 (+ 63us) server_negotiation.cc:300] Negotiation successful
0429 14:09:57.090064 (+ 28us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260429 14:09:57.091884 21027 raft_consensus.cc:1275] T e41b5b5b3a284cf19e626ff63ff24289 P 9b895bd9faeb41f99df00ab91689b6c2 [term 1 FOLLOWER]: Refusing update from remote peer 90a829bc5a9a4741acc14fb9c74d87cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:09:57.091915 20755 raft_consensus.cc:1275] T e41b5b5b3a284cf19e626ff63ff24289 P 21d7d2ed7d5b4cff9381c2f47a4b5ac4 [term 1 FOLLOWER]: Refusing update from remote peer 90a829bc5a9a4741acc14fb9c74d87cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:09:57.092326 21100 consensus_queue.cc:1048] T e41b5b5b3a284cf19e626ff63ff24289 P 90a829bc5a9a4741acc14fb9c74d87cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b895bd9faeb41f99df00ab91689b6c2" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 34651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:09:57.092486 21096 consensus_queue.cc:1048] T e41b5b5b3a284cf19e626ff63ff24289 P 90a829bc5a9a4741acc14fb9c74d87cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d7d2ed7d5b4cff9381c2f47a4b5ac4" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 35909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:09:57.094923 21103 mvcc.cc:204] Tried to move back new op lower bound from 7280524480885694464 to 7280524480752656384. Current Snapshot: MvccSnapshot[applied={T|T < 7280524480885694464}]
I20260429 14:09:57.095257 21105 mvcc.cc:204] Tried to move back new op lower bound from 7280524480885694464 to 7280524480752656384. Current Snapshot: MvccSnapshot[applied={T|T < 7280524480885694464}]
I20260429 14:09:57.095374 21104 mvcc.cc:204] Tried to move back new op lower bound from 7280524480885694464 to 7280524480752656384. Current Snapshot: MvccSnapshot[applied={T|T < 7280524480885694464}]
I20260429 14:09:57.097541 20606 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:09:57.097643 20606 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:09:57.098572 20606 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 00a450a851574a89965a4d5b8fd8d2a1: Sending DeleteTablet for 3 replicas of tablet e41b5b5b3a284cf19e626ff63ff24289
I20260429 14:09:57.098924 21007 tablet_service.cc:1558] Processing DeleteTablet for tablet e41b5b5b3a284cf19e626ff63ff24289 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:09:57 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:49326
I20260429 14:09:57.098996 20871 tablet_service.cc:1558] Processing DeleteTablet for tablet e41b5b5b3a284cf19e626ff63ff24289 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:09:57 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:36134
I20260429 14:09:57.099051 20735 tablet_service.cc:1558] Processing DeleteTablet for tablet e41b5b5b3a284cf19e626ff63ff24289 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:09:57 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:43962
I20260429 14:09:57.099318 21113 tablet_replica.cc:333] T e41b5b5b3a284cf19e626ff63ff24289 P 90a829bc5a9a4741acc14fb9c74d87cc: stopping tablet replica
I20260429 14:09:57.099319 21114 tablet_replica.cc:333] T e41b5b5b3a284cf19e626ff63ff24289 P 21d7d2ed7d5b4cff9381c2f47a4b5ac4: stopping tablet replica
I20260429 14:09:57.099404 21114 raft_consensus.cc:2243] T e41b5b5b3a284cf19e626ff63ff24289 P 21d7d2ed7d5b4cff9381c2f47a4b5ac4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:09:57.099436 21113 raft_consensus.cc:2243] T e41b5b5b3a284cf19e626ff63ff24289 P 90a829bc5a9a4741acc14fb9c74d87cc [term 1 LEADER]: Raft consensus shutting down.
I20260429 14:09:57.099551 21114 raft_consensus.cc:2272] T e41b5b5b3a284cf19e626ff63ff24289 P 21d7d2ed7d5b4cff9381c2f47a4b5ac4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:09:57.099551 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 20672
I20260429 14:09:57.100098 21113 raft_consensus.cc:2272] T e41b5b5b3a284cf19e626ff63ff24289 P 90a829bc5a9a4741acc14fb9c74d87cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:09:57.100592 21113 ts_tablet_manager.cc:1916] T e41b5b5b3a284cf19e626ff63ff24289 P 90a829bc5a9a4741acc14fb9c74d87cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:09:57.102159 21113 ts_tablet_manager.cc:1929] T e41b5b5b3a284cf19e626ff63ff24289 P 90a829bc5a9a4741acc14fb9c74d87cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 14:09:57.102442 21113 log.cc:1199] T e41b5b5b3a284cf19e626ff63ff24289 P 90a829bc5a9a4741acc14fb9c74d87cc: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-1/wal/wals/e41b5b5b3a284cf19e626ff63ff24289
I20260429 14:09:57.102785 21113 ts_tablet_manager.cc:1950] T e41b5b5b3a284cf19e626ff63ff24289 P 90a829bc5a9a4741acc14fb9c74d87cc: Deleting consensus metadata
I20260429 14:09:57.103430 20592 catalog_manager.cc:5002] TS 90a829bc5a9a4741acc14fb9c74d87cc (127.2.197.194:42309): tablet e41b5b5b3a284cf19e626ff63ff24289 (table test-table [id=3b64b8ea533d41e78baf725d74fa787d]) successfully deleted
I20260429 14:09:57.106482 21112 tablet_replica.cc:333] T e41b5b5b3a284cf19e626ff63ff24289 P 9b895bd9faeb41f99df00ab91689b6c2: stopping tablet replica
I20260429 14:09:57.106601 21112 raft_consensus.cc:2243] T e41b5b5b3a284cf19e626ff63ff24289 P 9b895bd9faeb41f99df00ab91689b6c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:09:57.106724 21112 raft_consensus.cc:2272] T e41b5b5b3a284cf19e626ff63ff24289 P 9b895bd9faeb41f99df00ab91689b6c2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260429 14:09:57.107025 20592 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.197.193:35909 (error 108)
I20260429 14:09:57.107336 21112 ts_tablet_manager.cc:1916] T e41b5b5b3a284cf19e626ff63ff24289 P 9b895bd9faeb41f99df00ab91689b6c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:09:57.107347 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 20808
I20260429 14:09:57.109133 21084 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:57.108964 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:35909 (local address 127.0.0.1:43974)
0429 14:09:57.109051 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:57.109094 (+ 43us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.193:35909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260429 14:09:57.109236 21112 ts_tablet_manager.cc:1929] T e41b5b5b3a284cf19e626ff63ff24289 P 9b895bd9faeb41f99df00ab91689b6c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 14:09:57.109349 21112 log.cc:1199] T e41b5b5b3a284cf19e626ff63ff24289 P 9b895bd9faeb41f99df00ab91689b6c2: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777471413505459-2839-0/minicluster-data/ts-2/wal/wals/e41b5b5b3a284cf19e626ff63ff24289
W20260429 14:09:57.109316 20592 catalog_manager.cc:4729] TS 21d7d2ed7d5b4cff9381c2f47a4b5ac4 (127.2.197.193:35909): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e41b5b5b3a284cf19e626ff63ff24289: Network error: Client connection negotiation failed: client connection to 127.2.197.193:35909: connect: Connection refused (error 111)
I20260429 14:09:57.109609 21112 ts_tablet_manager.cc:1950] T e41b5b5b3a284cf19e626ff63ff24289 P 9b895bd9faeb41f99df00ab91689b6c2: Deleting consensus metadata
I20260429 14:09:57.110013 20593 catalog_manager.cc:5002] TS 9b895bd9faeb41f99df00ab91689b6c2 (127.2.197.195:34651): tablet e41b5b5b3a284cf19e626ff63ff24289 (table test-table [id=3b64b8ea533d41e78baf725d74fa787d]) successfully deleted
I20260429 14:09:57.112581 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 20944
I20260429 14:09:57.117451 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 20576
2026-04-29T14:09:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (3152 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-04-29T14:09:57Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:09:57Z Disabled control of system clock
I20260429 14:09:57.145114 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:42811
--webserver_interface=127.2.197.254
--webserver_port=0
--builtin_ntp_servers=127.2.197.212:45409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:42811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260429 14:09:57.221447 21119 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:09:57.221629 21119 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:09:57.221652 21119 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:09:57.223156 21119 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:09:57.223201 21119 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:09:57.223214 21119 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:09:57.223227 21119 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:09:57.223238 21119 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:09:57.224721 21119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:45409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:42811
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:42811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:09:57.224915 21119 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:09:57.225091 21119 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:09:57.227535 21125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:09:57.227635 21127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:09:57.227700 21119 server_base.cc:1061] running on GCE node
W20260429 14:09:57.227634 21124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:09:57.228076 21119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:09:57.228360 21119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:09:57.229521 21119 hybrid_clock.cc:648] HybridClock initialized: now 1777471797229509 us; error 47 us; skew 500 ppm
I20260429 14:09:57.230700 21119 webserver.cc:492] Webserver started at http://127.2.197.254:41159/ using document root <none> and password file <none>
I20260429 14:09:57.230916 21119 fs_manager.cc:362] Metadata directory not provided
I20260429 14:09:57.230981 21119 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:09:57.231087 21119 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:09:57.232116 21119 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "1f92fd003a2a426da0265ead14a040ef"
format_stamp: "Formatted at 2026-04-29 14:09:57 on dist-test-slave-8n77"
server_key: "7e3c23a8f4ad55d83978044042376c4a"
server_key_iv: "93bc868d823b3eb4f2bab4cee8228c14"
server_key_version: "encryptionkey@0"
I20260429 14:09:57.232465 21119 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "1f92fd003a2a426da0265ead14a040ef"
format_stamp: "Formatted at 2026-04-29 14:09:57 on dist-test-slave-8n77"
server_key: "7e3c23a8f4ad55d83978044042376c4a"
server_key_iv: "93bc868d823b3eb4f2bab4cee8228c14"
server_key_version: "encryptionkey@0"
I20260429 14:09:57.233726 21119 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260429 14:09:57.234571 21133 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:09:57.234750 21119 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260429 14:09:57.234838 21119 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "1f92fd003a2a426da0265ead14a040ef"
format_stamp: "Formatted at 2026-04-29 14:09:57 on dist-test-slave-8n77"
server_key: "7e3c23a8f4ad55d83978044042376c4a"
server_key_iv: "93bc868d823b3eb4f2bab4cee8228c14"
server_key_version: "encryptionkey@0"
I20260429 14:09:57.234915 21119 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:09:57.249670 21119 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:09:57.249998 21119 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:09:57.250152 21119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:09:57.254029 21119 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:42811
I20260429 14:09:57.254055 21185 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:42811 every 8 connection(s)
I20260429 14:09:57.254442 21119 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:09:57.254983 21186 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:09:57.257377 21186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f92fd003a2a426da0265ead14a040ef: Bootstrap starting.
I20260429 14:09:57.257961 21186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f92fd003a2a426da0265ead14a040ef: Neither blocks nor log segments found. Creating new log.
I20260429 14:09:57.258222 21186 log.cc:826] T 00000000000000000000000000000000 P 1f92fd003a2a426da0265ead14a040ef: Log is configured to *not* fsync() on all Append() calls
I20260429 14:09:57.258929 21186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f92fd003a2a426da0265ead14a040ef: No bootstrap required, opened a new log
I20260429 14:09:57.259385 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 21119
I20260429 14:09:57.259514 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:09:57.259720 2839 external_mini_cluster.cc:1468] Setting key 54160982de877ff213522e6a681d4660
I20260429 14:09:57.260604 21186 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1f92fd003a2a426da0265ead14a040ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f92fd003a2a426da0265ead14a040ef" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 42811 } }
I20260429 14:09:57.260742 21186 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1f92fd003a2a426da0265ead14a040ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:09:57.260792 21186 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1f92fd003a2a426da0265ead14a040ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f92fd003a2a426da0265ead14a040ef, State: Initialized, Role: FOLLOWER
I20260429 14:09:57.260923 21186 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f92fd003a2a426da0265ead14a040ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f92fd003a2a426da0265ead14a040ef" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 42811 } }
I20260429 14:09:57.261003 21186 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1f92fd003a2a426da0265ead14a040ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:09:57.261034 21186 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1f92fd003a2a426da0265ead14a040ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:09:57.261078 21186 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1f92fd003a2a426da0265ead14a040ef [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:09:57.261659 21186 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1f92fd003a2a426da0265ead14a040ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f92fd003a2a426da0265ead14a040ef" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 42811 } }
I20260429 14:09:57.261797 21186 leader_election.cc:304] T 00000000000000000000000000000000 P 1f92fd003a2a426da0265ead14a040ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f92fd003a2a426da0265ead14a040ef; no voters:
I20260429 14:09:57.261973 21186 leader_election.cc:290] T 00000000000000000000000000000000 P 1f92fd003a2a426da0265ead14a040ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:09:57.262010 21191 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1f92fd003a2a426da0265ead14a040ef [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:09:57.262106 21191 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1f92fd003a2a426da0265ead14a040ef [term 1 LEADER]: Becoming Leader. State: Replica: 1f92fd003a2a426da0265ead14a040ef, State: Running, Role: LEADER
I20260429 14:09:57.262225 21186 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1f92fd003a2a426da0265ead14a040ef [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:09:57.262266 21191 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f92fd003a2a426da0265ead14a040ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f92fd003a2a426da0265ead14a040ef" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 42811 } }
I20260429 14:09:57.262631 21191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f92fd003a2a426da0265ead14a040ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f92fd003a2a426da0265ead14a040ef. Latest consensus state: current_term: 1 leader_uuid: "1f92fd003a2a426da0265ead14a040ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f92fd003a2a426da0265ead14a040ef" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 42811 } } }
I20260429 14:09:57.262639 21192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f92fd003a2a426da0265ead14a040ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f92fd003a2a426da0265ead14a040ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f92fd003a2a426da0265ead14a040ef" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 42811 } } }
I20260429 14:09:57.262815 21192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f92fd003a2a426da0265ead14a040ef [sys.catalog]: This master's current role is: LEADER
I20260429 14:09:57.262998 21191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f92fd003a2a426da0265ead14a040ef [sys.catalog]: This master's current role is: LEADER
I20260429 14:09:57.263175 21197 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:09:57.263957 21197 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:09:57.264786 21190 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:57.260242 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36184 (local address 127.2.197.254:42811)
0429 14:09:57.260467 (+ 225us) server_negotiation.cc:207] Beginning negotiation
0429 14:09:57.260476 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0429 14:09:57.260557 (+ 81us) server_negotiation.cc:408] Connection header received
0429 14:09:57.260675 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:09:57.260680 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:09:57.260721 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:09:57.260839 (+ 118us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:09:57.261330 (+ 491us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.262367 (+ 1037us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:09:57.263433 (+ 1066us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.263880 (+ 447us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:57.263982 (+ 102us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:09:57.263989 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:09:57.263990 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:09:57.264208 (+ 218us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:09:57.264228 (+ 20us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:09:57.264236 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:09:57.264555 (+ 319us) server_negotiation.cc:1036] Waiting for connection context
0429 14:09:57.264590 (+ 35us) server_negotiation.cc:300] Negotiation successful
0429 14:09:57.264639 (+ 49us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":121,"threads_started":1}
I20260429 14:09:57.265548 21197 catalog_manager.cc:1357] Generated new cluster ID: 0617b3e898e0478b87fcfee58f9067e0
I20260429 14:09:57.265604 21197 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:09:57.278359 21197 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:09:57.279000 21197 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:09:57.287191 21197 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 1f92fd003a2a426da0265ead14a040ef: Generated new TSK 0
I20260429 14:09:57.287371 21197 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 14:09:57.293263 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.193:0
--local_ip_for_outbound_sockets=127.2.197.193
--webserver_interface=127.2.197.193
--webserver_port=0
--tserver_master_addrs=127.2.197.254:42811
--builtin_ntp_servers=127.2.197.212:45409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260429 14:09:57.371783 21210 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:09:57.371965 21210 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:09:57.371989 21210 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:09:57.373584 21210 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:09:57.373639 21210 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:09:57.373667 21210 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.193
I20260429 14:09:57.375289 21210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:45409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.197.193
--webserver_port=0
--tserver_master_addrs=127.2.197.254:42811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.193
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:09:57.375530 21210 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:09:57.375777 21210 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:09:57.378597 21216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:09:57.378516 21215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:09:57.378499 21218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:09:57.378922 21210 server_base.cc:1061] running on GCE node
I20260429 14:09:57.379101 21210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:09:57.379354 21210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:09:57.380530 21210 hybrid_clock.cc:648] HybridClock initialized: now 1777471797380510 us; error 29 us; skew 500 ppm
I20260429 14:09:57.381695 21210 webserver.cc:492] Webserver started at http://127.2.197.193:43655/ using document root <none> and password file <none>
I20260429 14:09:57.381908 21210 fs_manager.cc:362] Metadata directory not provided
I20260429 14:09:57.381978 21210 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:09:57.382086 21210 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:09:57.383090 21210 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-0/data/instance:
uuid: "f234c404c77340919c97913d2a8ef231"
format_stamp: "Formatted at 2026-04-29 14:09:57 on dist-test-slave-8n77"
server_key: "5018be099f6a386cbaaac47fcc00a57b"
server_key_iv: "00b3bfc08cbbb0b72bd16ce38fbde126"
server_key_version: "encryptionkey@0"
I20260429 14:09:57.383430 21210 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance:
uuid: "f234c404c77340919c97913d2a8ef231"
format_stamp: "Formatted at 2026-04-29 14:09:57 on dist-test-slave-8n77"
server_key: "5018be099f6a386cbaaac47fcc00a57b"
server_key_iv: "00b3bfc08cbbb0b72bd16ce38fbde126"
server_key_version: "encryptionkey@0"
I20260429 14:09:57.384663 21210 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260429 14:09:57.385447 21224 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:09:57.385665 21210 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:09:57.385746 21210 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-0/wal
uuid: "f234c404c77340919c97913d2a8ef231"
format_stamp: "Formatted at 2026-04-29 14:09:57 on dist-test-slave-8n77"
server_key: "5018be099f6a386cbaaac47fcc00a57b"
server_key_iv: "00b3bfc08cbbb0b72bd16ce38fbde126"
server_key_version: "encryptionkey@0"
I20260429 14:09:57.385824 21210 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:09:57.404354 21210 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:09:57.404693 21210 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:09:57.404844 21210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:09:57.405114 21210 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:09:57.405504 21210 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:09:57.405562 21210 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:09:57.405603 21210 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:09:57.405632 21210 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:09:57.413010 21210 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.193:34319
I20260429 14:09:57.413084 21337 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.193:34319 every 8 connection(s)
I20260429 14:09:57.413405 21210 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
I20260429 14:09:57.418218 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 21210
I20260429 14:09:57.418392 21190 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:57.414153 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:37733 (local address 127.2.197.254:42811)
0429 14:09:57.414223 (+ 70us) server_negotiation.cc:207] Beginning negotiation
0429 14:09:57.414226 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:09:57.414574 (+ 348us) server_negotiation.cc:408] Connection header received
0429 14:09:57.415163 (+ 589us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:09:57.415165 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:09:57.415188 (+ 23us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:09:57.415290 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:09:57.415940 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.416453 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:09:57.417250 (+ 797us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.417410 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:57.417989 (+ 579us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:09:57.417993 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:09:57.417994 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:09:57.418002 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:09:57.418021 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:09:57.418025 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:09:57.418091 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0429 14:09:57.418190 (+ 99us) server_negotiation.cc:300] Negotiation successful
0429 14:09:57.418230 (+ 40us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260429 14:09:57.418459 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance
I20260429 14:09:57.418427 21340 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:57.414151 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:42811 (local address 127.2.197.193:37733)
0429 14:09:57.414357 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:57.414392 (+ 35us) client_negotiation.cc:175] Beginning negotiation
0429 14:09:57.414993 (+ 601us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:09:57.415373 (+ 380us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:09:57.415377 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:09:57.415388 (+ 11us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:09:57.415786 (+ 398us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:57.415789 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.416563 (+ 774us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:09:57.416564 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:09:57.417153 (+ 589us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:57.417156 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.417277 (+ 121us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:57.417844 (+ 567us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 14:09:57.417845 (+ 1us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:09:57.417858 (+ 13us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 14:09:57.417880 (+ 22us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:09:57.418101 (+ 221us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:09:57.418105 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:09:57.418107 (+ 2us) client_negotiation.cc:770] Sending connection context
0429 14:09:57.418174 (+ 67us) client_negotiation.cc:241] Negotiation successful
0429 14:09:57.418243 (+ 69us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":80,"threads_started":1}
I20260429 14:09:57.418705 2839 external_mini_cluster.cc:1468] Setting key 7a329423b54012469080ee55e62a8f51
I20260429 14:09:57.419030 21338 heartbeater.cc:344] Connected to a master server at 127.2.197.254:42811
I20260429 14:09:57.419142 21338 heartbeater.cc:461] Registering TS with master...
I20260429 14:09:57.419354 21338 heartbeater.cc:507] Master 127.2.197.254:42811 requested a full tablet report, sending...
I20260429 14:09:57.419651 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.194:0
--local_ip_for_outbound_sockets=127.2.197.194
--webserver_interface=127.2.197.194
--webserver_port=0
--tserver_master_addrs=127.2.197.254:42811
--builtin_ntp_servers=127.2.197.212:45409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260429 14:09:57.419849 21150 ts_manager.cc:194] Registered new tserver with Master: f234c404c77340919c97913d2a8ef231 (127.2.197.193:34319)
I20260429 14:09:57.420425 21150 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.2.197.193:37733
W20260429 14:09:57.500587 21341 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:09:57.500804 21341 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:09:57.500842 21341 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:09:57.502733 21341 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:09:57.502812 21341 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:09:57.502856 21341 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.194
I20260429 14:09:57.504550 21341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:45409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.197.194
--webserver_port=0
--tserver_master_addrs=127.2.197.254:42811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.194
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:09:57.504822 21341 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:09:57.505054 21341 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:09:57.507990 21349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:09:57.508000 21347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:09:57.508025 21346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:09:57.508036 21341 server_base.cc:1061] running on GCE node
I20260429 14:09:57.508543 21341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:09:57.508733 21341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:09:57.509924 21341 hybrid_clock.cc:648] HybridClock initialized: now 1777471797509896 us; error 47 us; skew 500 ppm
I20260429 14:09:57.511344 21341 webserver.cc:492] Webserver started at http://127.2.197.194:45513/ using document root <none> and password file <none>
I20260429 14:09:57.511667 21341 fs_manager.cc:362] Metadata directory not provided
I20260429 14:09:57.511725 21341 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:09:57.511837 21341 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:09:57.512835 21341 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-1/data/instance:
uuid: "ab39eafd963848519f6dda79aea95794"
format_stamp: "Formatted at 2026-04-29 14:09:57 on dist-test-slave-8n77"
server_key: "ee45c06723fbfc46a5a5c8e1428f1b47"
server_key_iv: "453885e9934d44ab86085708c9f347b0"
server_key_version: "encryptionkey@0"
I20260429 14:09:57.513171 21341 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance:
uuid: "ab39eafd963848519f6dda79aea95794"
format_stamp: "Formatted at 2026-04-29 14:09:57 on dist-test-slave-8n77"
server_key: "ee45c06723fbfc46a5a5c8e1428f1b47"
server_key_iv: "453885e9934d44ab86085708c9f347b0"
server_key_version: "encryptionkey@0"
I20260429 14:09:57.514462 21341 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260429 14:09:57.515239 21355 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:09:57.515586 21341 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:09:57.515645 21341 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-1/wal
uuid: "ab39eafd963848519f6dda79aea95794"
format_stamp: "Formatted at 2026-04-29 14:09:57 on dist-test-slave-8n77"
server_key: "ee45c06723fbfc46a5a5c8e1428f1b47"
server_key_iv: "453885e9934d44ab86085708c9f347b0"
server_key_version: "encryptionkey@0"
I20260429 14:09:57.515722 21341 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:09:57.524487 21341 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:09:57.524778 21341 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:09:57.524905 21341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:09:57.525116 21341 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:09:57.525465 21341 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:09:57.525521 21341 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:09:57.525557 21341 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:09:57.525604 21341 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:09:57.531515 21341 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.194:34319
I20260429 14:09:57.531620 21468 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.194:34319 every 8 connection(s)
I20260429 14:09:57.531912 21341 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
I20260429 14:09:57.534614 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 21341
I20260429 14:09:57.534740 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance
I20260429 14:09:57.534927 2839 external_mini_cluster.cc:1468] Setting key c46fea4d09d1d66c8f8fe2cb68a5316d
I20260429 14:09:57.536042 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.195:0
--local_ip_for_outbound_sockets=127.2.197.195
--webserver_interface=127.2.197.195
--webserver_port=0
--tserver_master_addrs=127.2.197.254:42811
--builtin_ntp_servers=127.2.197.212:45409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260429 14:09:57.536496 21190 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:57.532679 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:39743 (local address 127.2.197.254:42811)
0429 14:09:57.532732 (+ 53us) server_negotiation.cc:207] Beginning negotiation
0429 14:09:57.532734 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:09:57.533013 (+ 279us) server_negotiation.cc:408] Connection header received
0429 14:09:57.533595 (+ 582us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:09:57.533597 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:09:57.533608 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:09:57.533668 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:09:57.534102 (+ 434us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.534672 (+ 570us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:09:57.535363 (+ 691us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.535532 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:57.536199 (+ 667us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:09:57.536203 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:09:57.536203 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:09:57.536209 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:09:57.536224 (+ 15us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:09:57.536228 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:09:57.536277 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
0429 14:09:57.536379 (+ 102us) server_negotiation.cc:300] Negotiation successful
0429 14:09:57.536414 (+ 35us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260429 14:09:57.536628 21471 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:57.532680 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:42811 (local address 127.2.197.194:39743)
0429 14:09:57.532891 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:57.532920 (+ 29us) client_negotiation.cc:175] Beginning negotiation
0429 14:09:57.533446 (+ 526us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:09:57.533730 (+ 284us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:09:57.533734 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:09:57.533742 (+ 8us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:09:57.534026 (+ 284us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:57.534028 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.534767 (+ 739us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:09:57.534768 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:09:57.535237 (+ 469us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:57.535240 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.535415 (+ 175us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:57.536056 (+ 641us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 14:09:57.536056 (+ 0us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:09:57.536067 (+ 11us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 14:09:57.536085 (+ 18us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:09:57.536290 (+ 205us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:09:57.536295 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:09:57.536298 (+ 3us) client_negotiation.cc:770] Sending connection context
0429 14:09:57.536385 (+ 87us) client_negotiation.cc:241] Negotiation successful
0429 14:09:57.536454 (+ 69us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":80,"threads_started":1}
I20260429 14:09:57.537063 21469 heartbeater.cc:344] Connected to a master server at 127.2.197.254:42811
I20260429 14:09:57.537146 21469 heartbeater.cc:461] Registering TS with master...
I20260429 14:09:57.537395 21469 heartbeater.cc:507] Master 127.2.197.254:42811 requested a full tablet report, sending...
I20260429 14:09:57.537847 21150 ts_manager.cc:194] Registered new tserver with Master: ab39eafd963848519f6dda79aea95794 (127.2.197.194:34319)
I20260429 14:09:57.538219 21150 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.2.197.194:39743
W20260429 14:09:57.614125 21472 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:09:57.614406 21472 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:09:57.614435 21472 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:09:57.616014 21472 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:09:57.616061 21472 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:09:57.616089 21472 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.195
I20260429 14:09:57.617671 21472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:45409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.197.195
--webserver_port=0
--tserver_master_addrs=127.2.197.254:42811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.195
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:09:57.617879 21472 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:09:57.618196 21472 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:09:57.621304 21477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:09:57.621294 21478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:09:57.621299 21480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:09:57.621662 21472 server_base.cc:1061] running on GCE node
I20260429 14:09:57.621887 21472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:09:57.622124 21472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:09:57.623313 21472 hybrid_clock.cc:648] HybridClock initialized: now 1777471797623291 us; error 71 us; skew 500 ppm
I20260429 14:09:57.624550 21472 webserver.cc:492] Webserver started at http://127.2.197.195:39587/ using document root <none> and password file <none>
I20260429 14:09:57.624761 21472 fs_manager.cc:362] Metadata directory not provided
I20260429 14:09:57.624828 21472 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:09:57.624934 21472 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:09:57.625888 21472 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-2/data/instance:
uuid: "98f4c0a8f26b4070afe05990ca18f2bf"
format_stamp: "Formatted at 2026-04-29 14:09:57 on dist-test-slave-8n77"
server_key: "dff059ffa175786b7b3d4cb3b831803f"
server_key_iv: "5cc155a8f8cdfd8c179d7139fdf156aa"
server_key_version: "encryptionkey@0"
I20260429 14:09:57.626230 21472 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance:
uuid: "98f4c0a8f26b4070afe05990ca18f2bf"
format_stamp: "Formatted at 2026-04-29 14:09:57 on dist-test-slave-8n77"
server_key: "dff059ffa175786b7b3d4cb3b831803f"
server_key_iv: "5cc155a8f8cdfd8c179d7139fdf156aa"
server_key_version: "encryptionkey@0"
I20260429 14:09:57.627677 21472 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:09:57.628543 21486 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:09:57.628731 21472 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:09:57.628820 21472 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-2/wal
uuid: "98f4c0a8f26b4070afe05990ca18f2bf"
format_stamp: "Formatted at 2026-04-29 14:09:57 on dist-test-slave-8n77"
server_key: "dff059ffa175786b7b3d4cb3b831803f"
server_key_iv: "5cc155a8f8cdfd8c179d7139fdf156aa"
server_key_version: "encryptionkey@0"
I20260429 14:09:57.628901 21472 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:09:57.642858 21472 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:09:57.643312 21472 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:09:57.643494 21472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:09:57.643744 21472 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:09:57.644090 21472 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:09:57.644146 21472 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:09:57.644182 21472 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:09:57.644225 21472 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:09:57.650674 21472 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.195:39887
I20260429 14:09:57.650759 21599 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.195:39887 every 8 connection(s)
I20260429 14:09:57.651072 21472 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
I20260429 14:09:57.655728 21190 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:57.651923 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:54967 (local address 127.2.197.254:42811)
0429 14:09:57.651988 (+ 65us) server_negotiation.cc:207] Beginning negotiation
0429 14:09:57.651990 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:09:57.652202 (+ 212us) server_negotiation.cc:408] Connection header received
0429 14:09:57.652747 (+ 545us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:09:57.652748 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:09:57.652759 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:09:57.652829 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:09:57.653558 (+ 729us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.654040 (+ 482us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:09:57.654736 (+ 696us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.654874 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:57.655418 (+ 544us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:09:57.655422 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:09:57.655422 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:09:57.655428 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:09:57.655442 (+ 14us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:09:57.655446 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:09:57.655508 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0429 14:09:57.655633 (+ 125us) server_negotiation.cc:300] Negotiation successful
0429 14:09:57.655665 (+ 32us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260429 14:09:57.655826 21602 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:57.651874 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:42811 (local address 127.2.197.195:54967)
0429 14:09:57.652047 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:57.652078 (+ 31us) client_negotiation.cc:175] Beginning negotiation
0429 14:09:57.652534 (+ 456us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:09:57.652998 (+ 464us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:09:57.653003 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:09:57.653012 (+ 9us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:09:57.653433 (+ 421us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:57.653437 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.654152 (+ 715us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:09:57.654153 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:09:57.654635 (+ 482us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:57.654637 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.654737 (+ 100us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:57.655278 (+ 541us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 14:09:57.655279 (+ 1us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:09:57.655295 (+ 16us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 14:09:57.655315 (+ 20us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:09:57.655541 (+ 226us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:09:57.655546 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:09:57.655549 (+ 3us) client_negotiation.cc:770] Sending connection context
0429 14:09:57.655623 (+ 74us) client_negotiation.cc:241] Negotiation successful
0429 14:09:57.655684 (+ 61us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260429 14:09:57.656461 21600 heartbeater.cc:344] Connected to a master server at 127.2.197.254:42811
I20260429 14:09:57.656574 21600 heartbeater.cc:461] Registering TS with master...
I20260429 14:09:57.656783 21600 heartbeater.cc:507] Master 127.2.197.254:42811 requested a full tablet report, sending...
I20260429 14:09:57.657171 21150 ts_manager.cc:194] Registered new tserver with Master: 98f4c0a8f26b4070afe05990ca18f2bf (127.2.197.195:39887)
I20260429 14:09:57.657581 21150 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.2.197.195:54967
I20260429 14:09:57.661348 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 21472
I20260429 14:09:57.661485 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance
I20260429 14:09:57.661688 2839 external_mini_cluster.cc:1468] Setting key f5da73d58b5f524151176699921baa15
I20260429 14:09:57.663169 2839 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260429 14:09:57.667753 21190 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:57.665075 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36194 (local address 127.2.197.254:42811)
0429 14:09:57.665149 (+ 74us) server_negotiation.cc:207] Beginning negotiation
0429 14:09:57.665151 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:09:57.665213 (+ 62us) server_negotiation.cc:408] Connection header received
0429 14:09:57.665268 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:09:57.665268 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:09:57.665278 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:09:57.665324 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:09:57.665660 (+ 336us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.666133 (+ 473us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:09:57.666828 (+ 695us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.666963 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:57.667455 (+ 492us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:09:57.667458 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:09:57.667458 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:09:57.667463 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:09:57.667477 (+ 14us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:09:57.667480 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:09:57.667555 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0429 14:09:57.667660 (+ 105us) server_negotiation.cc:300] Negotiation successful
0429 14:09:57.667688 (+ 28us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260429 14:09:57.669461 21150 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 14:09:57.670809 21150 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 14:09:57.676222 21609 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:57.672835 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:34319 (local address 127.0.0.1:33684)
0429 14:09:57.673026 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:57.673059 (+ 33us) client_negotiation.cc:175] Beginning negotiation
0429 14:09:57.673135 (+ 76us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:09:57.673696 (+ 561us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:09:57.673697 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:09:57.673700 (+ 3us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:09:57.673916 (+ 216us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:57.673918 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.675172 (+ 1254us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:09:57.675173 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:09:57.675927 (+ 754us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:57.675929 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.675997 (+ 68us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:57.676032 (+ 35us) client_negotiation.cc:770] Sending connection context
0429 14:09:57.676080 (+ 48us) client_negotiation.cc:241] Negotiation successful
0429 14:09:57.676121 (+ 41us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":85,"threads_started":1}
I20260429 14:09:57.676726 21608 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:57.672860 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:34319 (local address 127.0.0.1:49602)
0429 14:09:57.673076 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:57.673087 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 14:09:57.673135 (+ 48us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:09:57.673473 (+ 338us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:09:57.673476 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:09:57.673479 (+ 3us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:09:57.673720 (+ 241us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:57.673722 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.675619 (+ 1897us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:09:57.675620 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:09:57.676506 (+ 886us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:57.676509 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.676590 (+ 81us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:57.676594 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 14:09:57.676627 (+ 33us) client_negotiation.cc:241] Negotiation successful
0429 14:09:57.676665 (+ 38us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":91,"threads_started":1}
I20260429 14:09:57.677387 21610 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:57.672932 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33684 (local address 127.2.197.193:34319)
0429 14:09:57.673549 (+ 617us) server_negotiation.cc:207] Beginning negotiation
0429 14:09:57.673554 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 14:09:57.673561 (+ 7us) server_negotiation.cc:408] Connection header received
0429 14:09:57.673591 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:09:57.673593 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:09:57.673603 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:09:57.673678 (+ 75us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:09:57.673994 (+ 316us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.675086 (+ 1092us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:09:57.676634 (+ 1548us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.677143 (+ 509us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:57.677226 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0429 14:09:57.677257 (+ 31us) server_negotiation.cc:300] Negotiation successful
0429 14:09:57.677309 (+ 52us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":77,"threads_started":1}
I20260429 14:09:57.677587 21613 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:57.673278 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:50612 (local address 127.2.197.195:39887)
0429 14:09:57.673504 (+ 226us) server_negotiation.cc:207] Beginning negotiation
0429 14:09:57.673511 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 14:09:57.673521 (+ 10us) server_negotiation.cc:408] Connection header received
0429 14:09:57.673573 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:09:57.673575 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:09:57.673589 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:09:57.673808 (+ 219us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:09:57.674170 (+ 362us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.675648 (+ 1478us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:09:57.676989 (+ 1341us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.677373 (+ 384us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:57.677419 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
0429 14:09:57.677474 (+ 55us) server_negotiation.cc:300] Negotiation successful
0429 14:09:57.677518 (+ 44us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":91,"threads_started":1}
I20260429 14:09:57.677645 21611 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:57.672942 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49602 (local address 127.2.197.194:34319)
0429 14:09:57.673260 (+ 318us) server_negotiation.cc:207] Beginning negotiation
0429 14:09:57.673267 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 14:09:57.673277 (+ 10us) server_negotiation.cc:408] Connection header received
0429 14:09:57.673315 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:09:57.673317 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:09:57.673333 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:09:57.673409 (+ 76us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:09:57.673884 (+ 475us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.675527 (+ 1643us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:09:57.676930 (+ 1403us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.677410 (+ 480us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:57.677481 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0429 14:09:57.677515 (+ 34us) server_negotiation.cc:300] Negotiation successful
0429 14:09:57.677566 (+ 51us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":76,"threads_started":1}
I20260429 14:09:57.676930 21612 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:57.673053 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:39887 (local address 127.0.0.1:50612)
0429 14:09:57.673181 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:57.673192 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 14:09:57.673254 (+ 62us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:09:57.673712 (+ 458us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:09:57.673713 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:09:57.673715 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:09:57.674104 (+ 389us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:57.674106 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.675730 (+ 1624us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:09:57.675731 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:09:57.676707 (+ 976us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:57.676710 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.676783 (+ 73us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:57.676786 (+ 3us) client_negotiation.cc:770] Sending connection context
0429 14:09:57.676816 (+ 30us) client_negotiation.cc:241] Negotiation successful
0429 14:09:57.676857 (+ 41us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260429 14:09:57.677943 21272 tablet_service.cc:1511] Processing CreateTablet for tablet 53d540ba16b34ede85c0b8c9abf7f80c (DEFAULT_TABLE table=test-table [id=51583aaf6f2e4c75bf7575806daa04d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:09:57.678182 21272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53d540ba16b34ede85c0b8c9abf7f80c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:09:57.680279 21614 tablet_bootstrap.cc:492] T 53d540ba16b34ede85c0b8c9abf7f80c P f234c404c77340919c97913d2a8ef231: Bootstrap starting.
I20260429 14:09:57.680722 21403 tablet_service.cc:1511] Processing CreateTablet for tablet 53d540ba16b34ede85c0b8c9abf7f80c (DEFAULT_TABLE table=test-table [id=51583aaf6f2e4c75bf7575806daa04d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:09:57.680960 21614 tablet_bootstrap.cc:654] T 53d540ba16b34ede85c0b8c9abf7f80c P f234c404c77340919c97913d2a8ef231: Neither blocks nor log segments found. Creating new log.
I20260429 14:09:57.680986 21403 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53d540ba16b34ede85c0b8c9abf7f80c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:09:57.680912 21534 tablet_service.cc:1511] Processing CreateTablet for tablet 53d540ba16b34ede85c0b8c9abf7f80c (DEFAULT_TABLE table=test-table [id=51583aaf6f2e4c75bf7575806daa04d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:09:57.681167 21534 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53d540ba16b34ede85c0b8c9abf7f80c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:09:57.681249 21614 log.cc:826] T 53d540ba16b34ede85c0b8c9abf7f80c P f234c404c77340919c97913d2a8ef231: Log is configured to *not* fsync() on all Append() calls
I20260429 14:09:57.682003 21614 tablet_bootstrap.cc:492] T 53d540ba16b34ede85c0b8c9abf7f80c P f234c404c77340919c97913d2a8ef231: No bootstrap required, opened a new log
I20260429 14:09:57.682060 21614 ts_tablet_manager.cc:1403] T 53d540ba16b34ede85c0b8c9abf7f80c P f234c404c77340919c97913d2a8ef231: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260429 14:09:57.683334 21616 tablet_bootstrap.cc:492] T 53d540ba16b34ede85c0b8c9abf7f80c P 98f4c0a8f26b4070afe05990ca18f2bf: Bootstrap starting.
I20260429 14:09:57.683758 21614 raft_consensus.cc:359] T 53d540ba16b34ede85c0b8c9abf7f80c P f234c404c77340919c97913d2a8ef231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f234c404c77340919c97913d2a8ef231" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34319 } } peers { permanent_uuid: "ab39eafd963848519f6dda79aea95794" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 34319 } } peers { permanent_uuid: "98f4c0a8f26b4070afe05990ca18f2bf" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 39887 } }
I20260429 14:09:57.683909 21614 raft_consensus.cc:385] T 53d540ba16b34ede85c0b8c9abf7f80c P f234c404c77340919c97913d2a8ef231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:09:57.683936 21614 raft_consensus.cc:740] T 53d540ba16b34ede85c0b8c9abf7f80c P f234c404c77340919c97913d2a8ef231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f234c404c77340919c97913d2a8ef231, State: Initialized, Role: FOLLOWER
I20260429 14:09:57.683964 21616 tablet_bootstrap.cc:654] T 53d540ba16b34ede85c0b8c9abf7f80c P 98f4c0a8f26b4070afe05990ca18f2bf: Neither blocks nor log segments found. Creating new log.
I20260429 14:09:57.684062 21614 consensus_queue.cc:260] T 53d540ba16b34ede85c0b8c9abf7f80c P f234c404c77340919c97913d2a8ef231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f234c404c77340919c97913d2a8ef231" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34319 } } peers { permanent_uuid: "ab39eafd963848519f6dda79aea95794" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 34319 } } peers { permanent_uuid: "98f4c0a8f26b4070afe05990ca18f2bf" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 39887 } }
I20260429 14:09:57.684253 21616 log.cc:826] T 53d540ba16b34ede85c0b8c9abf7f80c P 98f4c0a8f26b4070afe05990ca18f2bf: Log is configured to *not* fsync() on all Append() calls
I20260429 14:09:57.684327 21614 ts_tablet_manager.cc:1434] T 53d540ba16b34ede85c0b8c9abf7f80c P f234c404c77340919c97913d2a8ef231: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260429 14:09:57.684376 21338 heartbeater.cc:499] Master 127.2.197.254:42811 was elected leader, sending a full tablet report...
I20260429 14:09:57.684525 21617 tablet_bootstrap.cc:492] T 53d540ba16b34ede85c0b8c9abf7f80c P ab39eafd963848519f6dda79aea95794: Bootstrap starting.
I20260429 14:09:57.685099 21617 tablet_bootstrap.cc:654] T 53d540ba16b34ede85c0b8c9abf7f80c P ab39eafd963848519f6dda79aea95794: Neither blocks nor log segments found. Creating new log.
I20260429 14:09:57.685371 21617 log.cc:826] T 53d540ba16b34ede85c0b8c9abf7f80c P ab39eafd963848519f6dda79aea95794: Log is configured to *not* fsync() on all Append() calls
I20260429 14:09:57.686159 21617 tablet_bootstrap.cc:492] T 53d540ba16b34ede85c0b8c9abf7f80c P ab39eafd963848519f6dda79aea95794: No bootstrap required, opened a new log
I20260429 14:09:57.686167 21616 tablet_bootstrap.cc:492] T 53d540ba16b34ede85c0b8c9abf7f80c P 98f4c0a8f26b4070afe05990ca18f2bf: No bootstrap required, opened a new log
I20260429 14:09:57.686232 21617 ts_tablet_manager.cc:1403] T 53d540ba16b34ede85c0b8c9abf7f80c P ab39eafd963848519f6dda79aea95794: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260429 14:09:57.686232 21616 ts_tablet_manager.cc:1403] T 53d540ba16b34ede85c0b8c9abf7f80c P 98f4c0a8f26b4070afe05990ca18f2bf: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260429 14:09:57.687835 21616 raft_consensus.cc:359] T 53d540ba16b34ede85c0b8c9abf7f80c P 98f4c0a8f26b4070afe05990ca18f2bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f234c404c77340919c97913d2a8ef231" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34319 } } peers { permanent_uuid: "ab39eafd963848519f6dda79aea95794" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 34319 } } peers { permanent_uuid: "98f4c0a8f26b4070afe05990ca18f2bf" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 39887 } }
I20260429 14:09:57.687824 21617 raft_consensus.cc:359] T 53d540ba16b34ede85c0b8c9abf7f80c P ab39eafd963848519f6dda79aea95794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f234c404c77340919c97913d2a8ef231" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34319 } } peers { permanent_uuid: "ab39eafd963848519f6dda79aea95794" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 34319 } } peers { permanent_uuid: "98f4c0a8f26b4070afe05990ca18f2bf" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 39887 } }
I20260429 14:09:57.687976 21617 raft_consensus.cc:385] T 53d540ba16b34ede85c0b8c9abf7f80c P ab39eafd963848519f6dda79aea95794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:09:57.687975 21616 raft_consensus.cc:385] T 53d540ba16b34ede85c0b8c9abf7f80c P 98f4c0a8f26b4070afe05990ca18f2bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:09:57.688009 21616 raft_consensus.cc:740] T 53d540ba16b34ede85c0b8c9abf7f80c P 98f4c0a8f26b4070afe05990ca18f2bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f4c0a8f26b4070afe05990ca18f2bf, State: Initialized, Role: FOLLOWER
I20260429 14:09:57.688009 21617 raft_consensus.cc:740] T 53d540ba16b34ede85c0b8c9abf7f80c P ab39eafd963848519f6dda79aea95794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab39eafd963848519f6dda79aea95794, State: Initialized, Role: FOLLOWER
I20260429 14:09:57.688099 21616 consensus_queue.cc:260] T 53d540ba16b34ede85c0b8c9abf7f80c P 98f4c0a8f26b4070afe05990ca18f2bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f234c404c77340919c97913d2a8ef231" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34319 } } peers { permanent_uuid: "ab39eafd963848519f6dda79aea95794" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 34319 } } peers { permanent_uuid: "98f4c0a8f26b4070afe05990ca18f2bf" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 39887 } }
I20260429 14:09:57.688153 21617 consensus_queue.cc:260] T 53d540ba16b34ede85c0b8c9abf7f80c P ab39eafd963848519f6dda79aea95794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f234c404c77340919c97913d2a8ef231" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34319 } } peers { permanent_uuid: "ab39eafd963848519f6dda79aea95794" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 34319 } } peers { permanent_uuid: "98f4c0a8f26b4070afe05990ca18f2bf" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 39887 } }
I20260429 14:09:57.688378 21617 ts_tablet_manager.cc:1434] T 53d540ba16b34ede85c0b8c9abf7f80c P ab39eafd963848519f6dda79aea95794: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260429 14:09:57.688371 21616 ts_tablet_manager.cc:1434] T 53d540ba16b34ede85c0b8c9abf7f80c P 98f4c0a8f26b4070afe05990ca18f2bf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260429 14:09:57.688472 21469 heartbeater.cc:499] Master 127.2.197.254:42811 was elected leader, sending a full tablet report...
I20260429 14:09:57.688499 21600 heartbeater.cc:499] Master 127.2.197.254:42811 was elected leader, sending a full tablet report...
I20260429 14:09:57.715894 21622 raft_consensus.cc:493] T 53d540ba16b34ede85c0b8c9abf7f80c P ab39eafd963848519f6dda79aea95794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:09:57.716022 21622 raft_consensus.cc:515] T 53d540ba16b34ede85c0b8c9abf7f80c P ab39eafd963848519f6dda79aea95794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f234c404c77340919c97913d2a8ef231" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34319 } } peers { permanent_uuid: "ab39eafd963848519f6dda79aea95794" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 34319 } } peers { permanent_uuid: "98f4c0a8f26b4070afe05990ca18f2bf" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 39887 } }
I20260429 14:09:57.716426 21622 leader_election.cc:290] T 53d540ba16b34ede85c0b8c9abf7f80c P ab39eafd963848519f6dda79aea95794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f234c404c77340919c97913d2a8ef231 (127.2.197.193:34319), 98f4c0a8f26b4070afe05990ca18f2bf (127.2.197.195:39887)
I20260429 14:09:57.719604 21471 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:57.716701 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:34319 (local address 127.2.197.194:45395)
0429 14:09:57.716799 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:57.716815 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 14:09:57.716875 (+ 60us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:09:57.717177 (+ 302us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:09:57.717178 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:09:57.717181 (+ 3us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:09:57.717490 (+ 309us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:57.717495 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.718429 (+ 934us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:09:57.718430 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:09:57.719350 (+ 920us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:57.719352 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.719431 (+ 79us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:57.719436 (+ 5us) client_negotiation.cc:770] Sending connection context
0429 14:09:57.719469 (+ 33us) client_negotiation.cc:241] Negotiation successful
0429 14:09:57.719507 (+ 38us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260429 14:09:57.720077 21610 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:57.716914 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:45395 (local address 127.2.197.193:34319)
0429 14:09:57.716985 (+ 71us) server_negotiation.cc:207] Beginning negotiation
0429 14:09:57.716988 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:09:57.716998 (+ 10us) server_negotiation.cc:408] Connection header received
0429 14:09:57.717029 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:09:57.717030 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:09:57.717044 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:09:57.717114 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:09:57.717640 (+ 526us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.718347 (+ 707us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:09:57.719513 (+ 1166us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.719976 (+ 463us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:57.719984 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0429 14:09:57.720004 (+ 20us) server_negotiation.cc:300] Negotiation successful
0429 14:09:57.720030 (+ 26us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260429 14:09:57.720151 21623 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:57.716890 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:39887 (local address 127.2.197.194:47211)
0429 14:09:57.717198 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:57.717264 (+ 66us) client_negotiation.cc:175] Beginning negotiation
0429 14:09:57.717345 (+ 81us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:09:57.717530 (+ 185us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:09:57.717531 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:09:57.717534 (+ 3us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:09:57.717908 (+ 374us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:57.717910 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.718801 (+ 891us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:09:57.718802 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:09:57.719921 (+ 1119us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:09:57.719923 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.720007 (+ 84us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:57.720011 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 14:09:57.720042 (+ 31us) client_negotiation.cc:241] Negotiation successful
0429 14:09:57.720077 (+ 35us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":129,"threads_started":1}
I20260429 14:09:57.720638 21613 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:57.717095 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:47211 (local address 127.2.197.195:39887)
0429 14:09:57.717139 (+ 44us) server_negotiation.cc:207] Beginning negotiation
0429 14:09:57.717141 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:09:57.717370 (+ 229us) server_negotiation.cc:408] Connection header received
0429 14:09:57.717397 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:09:57.717398 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:09:57.717409 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:09:57.717458 (+ 49us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:09:57.718008 (+ 550us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.718698 (+ 690us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:09:57.720007 (+ 1309us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.720490 (+ 483us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:57.720500 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
0429 14:09:57.720543 (+ 43us) server_negotiation.cc:300] Negotiation successful
0429 14:09:57.720581 (+ 38us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260429 14:09:57.720574 21292 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53d540ba16b34ede85c0b8c9abf7f80c" candidate_uuid: "ab39eafd963848519f6dda79aea95794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f234c404c77340919c97913d2a8ef231" is_pre_election: true
I20260429 14:09:57.720760 21292 raft_consensus.cc:2468] T 53d540ba16b34ede85c0b8c9abf7f80c P f234c404c77340919c97913d2a8ef231 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab39eafd963848519f6dda79aea95794 in term 0.
I20260429 14:09:57.721037 21357 leader_election.cc:304] T 53d540ba16b34ede85c0b8c9abf7f80c P ab39eafd963848519f6dda79aea95794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab39eafd963848519f6dda79aea95794, f234c404c77340919c97913d2a8ef231; no voters:
I20260429 14:09:57.721105 21554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53d540ba16b34ede85c0b8c9abf7f80c" candidate_uuid: "ab39eafd963848519f6dda79aea95794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f4c0a8f26b4070afe05990ca18f2bf" is_pre_election: true
I20260429 14:09:57.721217 21622 raft_consensus.cc:2804] T 53d540ba16b34ede85c0b8c9abf7f80c P ab39eafd963848519f6dda79aea95794 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 14:09:57.721261 21554 raft_consensus.cc:2468] T 53d540ba16b34ede85c0b8c9abf7f80c P 98f4c0a8f26b4070afe05990ca18f2bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab39eafd963848519f6dda79aea95794 in term 0.
I20260429 14:09:57.721274 21622 raft_consensus.cc:493] T 53d540ba16b34ede85c0b8c9abf7f80c P ab39eafd963848519f6dda79aea95794 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 14:09:57.721306 21622 raft_consensus.cc:3060] T 53d540ba16b34ede85c0b8c9abf7f80c P ab39eafd963848519f6dda79aea95794 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:09:57.722204 21622 raft_consensus.cc:515] T 53d540ba16b34ede85c0b8c9abf7f80c P ab39eafd963848519f6dda79aea95794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f234c404c77340919c97913d2a8ef231" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34319 } } peers { permanent_uuid: "ab39eafd963848519f6dda79aea95794" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 34319 } } peers { permanent_uuid: "98f4c0a8f26b4070afe05990ca18f2bf" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 39887 } }
I20260429 14:09:57.722400 21622 leader_election.cc:290] T 53d540ba16b34ede85c0b8c9abf7f80c P ab39eafd963848519f6dda79aea95794 [CANDIDATE]: Term 1 election: Requested vote from peers f234c404c77340919c97913d2a8ef231 (127.2.197.193:34319), 98f4c0a8f26b4070afe05990ca18f2bf (127.2.197.195:39887)
I20260429 14:09:57.722642 21292 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53d540ba16b34ede85c0b8c9abf7f80c" candidate_uuid: "ab39eafd963848519f6dda79aea95794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f234c404c77340919c97913d2a8ef231"
I20260429 14:09:57.722741 21554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53d540ba16b34ede85c0b8c9abf7f80c" candidate_uuid: "ab39eafd963848519f6dda79aea95794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f4c0a8f26b4070afe05990ca18f2bf"
I20260429 14:09:57.722754 21292 raft_consensus.cc:3060] T 53d540ba16b34ede85c0b8c9abf7f80c P f234c404c77340919c97913d2a8ef231 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:09:57.722811 21554 raft_consensus.cc:3060] T 53d540ba16b34ede85c0b8c9abf7f80c P 98f4c0a8f26b4070afe05990ca18f2bf [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:09:57.723694 21554 raft_consensus.cc:2468] T 53d540ba16b34ede85c0b8c9abf7f80c P 98f4c0a8f26b4070afe05990ca18f2bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab39eafd963848519f6dda79aea95794 in term 1.
I20260429 14:09:57.723694 21292 raft_consensus.cc:2468] T 53d540ba16b34ede85c0b8c9abf7f80c P f234c404c77340919c97913d2a8ef231 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab39eafd963848519f6dda79aea95794 in term 1.
I20260429 14:09:57.723906 21357 leader_election.cc:304] T 53d540ba16b34ede85c0b8c9abf7f80c P ab39eafd963848519f6dda79aea95794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab39eafd963848519f6dda79aea95794, f234c404c77340919c97913d2a8ef231; no voters:
I20260429 14:09:57.724030 21622 raft_consensus.cc:2804] T 53d540ba16b34ede85c0b8c9abf7f80c P ab39eafd963848519f6dda79aea95794 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:09:57.724231 21622 raft_consensus.cc:697] T 53d540ba16b34ede85c0b8c9abf7f80c P ab39eafd963848519f6dda79aea95794 [term 1 LEADER]: Becoming Leader. State: Replica: ab39eafd963848519f6dda79aea95794, State: Running, Role: LEADER
I20260429 14:09:57.724395 21622 consensus_queue.cc:237] T 53d540ba16b34ede85c0b8c9abf7f80c P ab39eafd963848519f6dda79aea95794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f234c404c77340919c97913d2a8ef231" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34319 } } peers { permanent_uuid: "ab39eafd963848519f6dda79aea95794" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 34319 } } peers { permanent_uuid: "98f4c0a8f26b4070afe05990ca18f2bf" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 39887 } }
I20260429 14:09:57.725237 21150 catalog_manager.cc:5671] T 53d540ba16b34ede85c0b8c9abf7f80c P ab39eafd963848519f6dda79aea95794 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab39eafd963848519f6dda79aea95794 (127.2.197.194). New cstate: current_term: 1 leader_uuid: "ab39eafd963848519f6dda79aea95794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f234c404c77340919c97913d2a8ef231" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab39eafd963848519f6dda79aea95794" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98f4c0a8f26b4070afe05990ca18f2bf" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 39887 } health_report { overall_health: UNKNOWN } } }
I20260429 14:09:57.738912 21611 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:57.736212 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49612 (local address 127.2.197.194:34319)
0429 14:09:57.736296 (+ 84us) server_negotiation.cc:207] Beginning negotiation
0429 14:09:57.736304 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0429 14:09:57.736313 (+ 9us) server_negotiation.cc:408] Connection header received
0429 14:09:57.736356 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:09:57.736357 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:09:57.736377 (+ 20us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:09:57.736518 (+ 141us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:09:57.736813 (+ 295us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.737407 (+ 594us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:09:57.738173 (+ 766us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:09:57.738435 (+ 262us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:09:57.738478 (+ 43us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:09:57.738604 (+ 126us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:09:57.738702 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0429 14:09:57.738776 (+ 74us) server_negotiation.cc:300] Negotiation successful
0429 14:09:57.738809 (+ 33us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260429 14:09:57.740839 21554 raft_consensus.cc:1275] T 53d540ba16b34ede85c0b8c9abf7f80c P 98f4c0a8f26b4070afe05990ca18f2bf [term 1 FOLLOWER]: Refusing update from remote peer ab39eafd963848519f6dda79aea95794: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:09:57.740839 21292 raft_consensus.cc:1275] T 53d540ba16b34ede85c0b8c9abf7f80c P f234c404c77340919c97913d2a8ef231 [term 1 FOLLOWER]: Refusing update from remote peer ab39eafd963848519f6dda79aea95794: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:09:57.741220 21624 consensus_queue.cc:1048] T 53d540ba16b34ede85c0b8c9abf7f80c P ab39eafd963848519f6dda79aea95794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f234c404c77340919c97913d2a8ef231" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:09:57.741344 21622 consensus_queue.cc:1048] T 53d540ba16b34ede85c0b8c9abf7f80c P ab39eafd963848519f6dda79aea95794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f4c0a8f26b4070afe05990ca18f2bf" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 39887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:09:57.743796 21627 mvcc.cc:204] Tried to move back new op lower bound from 7280524483543285760 to 7280524483479953408. Current Snapshot: MvccSnapshot[applied={T|T < 7280524483543285760}]
I20260429 14:09:57.743948 21629 mvcc.cc:204] Tried to move back new op lower bound from 7280524483543285760 to 7280524483479953408. Current Snapshot: MvccSnapshot[applied={T|T < 7280524483543285760}]
I20260429 14:09:57.745018 21628 mvcc.cc:204] Tried to move back new op lower bound from 7280524483543285760 to 7280524483479953408. Current Snapshot: MvccSnapshot[applied={T|T < 7280524483543285760}]
I20260429 14:09:57.746371 21150 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:09:57.746477 21150 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:09:57.747206 21150 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 1f92fd003a2a426da0265ead14a040ef: Sending DeleteTablet for 3 replicas of tablet 53d540ba16b34ede85c0b8c9abf7f80c
I20260429 14:09:57.747578 21534 tablet_service.cc:1558] Processing DeleteTablet for tablet 53d540ba16b34ede85c0b8c9abf7f80c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:09:57 UTC) from {username='slave'} at 127.0.0.1:50612
I20260429 14:09:57.747639 21272 tablet_service.cc:1558] Processing DeleteTablet for tablet 53d540ba16b34ede85c0b8c9abf7f80c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:09:57 UTC) from {username='slave'} at 127.0.0.1:33684
I20260429 14:09:57.747908 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 21210
I20260429 14:09:57.747923 21637 tablet_replica.cc:333] T 53d540ba16b34ede85c0b8c9abf7f80c P 98f4c0a8f26b4070afe05990ca18f2bf: stopping tablet replica
I20260429 14:09:57.748003 21637 raft_consensus.cc:2243] T 53d540ba16b34ede85c0b8c9abf7f80c P 98f4c0a8f26b4070afe05990ca18f2bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:09:57.748091 21637 raft_consensus.cc:2272] T 53d540ba16b34ede85c0b8c9abf7f80c P 98f4c0a8f26b4070afe05990ca18f2bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:09:57.750356 21637 ts_tablet_manager.cc:1916] T 53d540ba16b34ede85c0b8c9abf7f80c P 98f4c0a8f26b4070afe05990ca18f2bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:09:57.750330 21403 tablet_service.cc:1558] Processing DeleteTablet for tablet 53d540ba16b34ede85c0b8c9abf7f80c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:09:57 UTC) from {username='slave'} at 127.0.0.1:49602
I20260429 14:09:57.751946 21637 ts_tablet_manager.cc:1929] T 53d540ba16b34ede85c0b8c9abf7f80c P 98f4c0a8f26b4070afe05990ca18f2bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 14:09:57.752027 21637 log.cc:1199] T 53d540ba16b34ede85c0b8c9abf7f80c P 98f4c0a8f26b4070afe05990ca18f2bf: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777471413505459-2839-0/minicluster-data/ts-2/wal/wals/53d540ba16b34ede85c0b8c9abf7f80c
I20260429 14:09:57.752354 21637 ts_tablet_manager.cc:1950] T 53d540ba16b34ede85c0b8c9abf7f80c P 98f4c0a8f26b4070afe05990ca18f2bf: Deleting consensus metadata
I20260429 14:09:57.752760 21135 catalog_manager.cc:5002] TS 98f4c0a8f26b4070afe05990ca18f2bf (127.2.197.195:39887): tablet 53d540ba16b34ede85c0b8c9abf7f80c (table test-table [id=51583aaf6f2e4c75bf7575806daa04d5]) successfully deleted
I20260429 14:09:57.754449 21639 tablet_replica.cc:333] T 53d540ba16b34ede85c0b8c9abf7f80c P ab39eafd963848519f6dda79aea95794: stopping tablet replica
I20260429 14:09:57.754585 21639 raft_consensus.cc:2243] T 53d540ba16b34ede85c0b8c9abf7f80c P ab39eafd963848519f6dda79aea95794 [term 1 LEADER]: Raft consensus shutting down.
I20260429 14:09:57.754726 21639 raft_consensus.cc:2272] T 53d540ba16b34ede85c0b8c9abf7f80c P ab39eafd963848519f6dda79aea95794 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260429 14:09:57.754771 21135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.197.193:34319 (error 108)
I20260429 14:09:57.754977 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 21341
I20260429 14:09:57.755759 21608 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:57.755287 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:34319 (local address 127.0.0.1:33698)
0429 14:09:57.755678 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:57.755715 (+ 37us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.193:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370}
W20260429 14:09:57.756392 21135 catalog_manager.cc:4729] TS f234c404c77340919c97913d2a8ef231 (127.2.197.193:34319): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 53d540ba16b34ede85c0b8c9abf7f80c: Network error: Client connection negotiation failed: client connection to 127.2.197.193:34319: connect: Connection refused (error 111)
I20260429 14:09:57.760454 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 21472
I20260429 14:09:57.761401 21608 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:09:57.760792 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:34319 (local address 127.0.0.1:49620)
0429 14:09:57.761311 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0429 14:09:57.761350 (+ 39us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.194:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494}
I20260429 14:09:57.766098 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 21119
2026-04-29T14:09:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (648 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:09:57 dist-test-slave-8n77 krb5kdc[21643](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:09:57 dist-test-slave-8n77 krb5kdc[21643](info): set up 2 sockets
Apr 29 14:09:57 dist-test-slave-8n77 krb5kdc[21643](info): commencing operation
krb5kdc: starting...
W20260429 14:09:59.787262 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.001s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:09:59 dist-test-slave-8n77 krb5kdc[21643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:09:59Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:09:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:09:59.938484 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:40915
--webserver_interface=127.2.197.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:35379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:40915
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:10:00.018796 21659 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:00.019026 21659 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:00.019053 21659 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:10:00.020808 21659 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:10:00.020881 21659 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:00.020897 21659 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:00.020911 21659 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:10:00.020923 21659 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:10:00.022933 21659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:35379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:40915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:40915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:00.023226 21659 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:00.023459 21659 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:10:00.027220 21665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:00.027225 21667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:00.027242 21659 server_base.cc:1061] running on GCE node
W20260429 14:10:00.027225 21664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:00.028218 21659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:00.028517 21659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:00.029968 21659 hybrid_clock.cc:648] HybridClock initialized: now 1777471800029925 us; error 53 us; skew 500 ppm
Apr 29 14:10:00 dist-test-slave-8n77 krb5kdc[21643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471800, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:10:00.032888 21659 init.cc:377] Logged in from keytab as kudu/127.2.197.254@KRBTEST.COM (short username kudu)
I20260429 14:10:00.033529 21659 webserver.cc:492] Webserver started at http://127.2.197.254:33785/ using document root <none> and password file <none>
I20260429 14:10:00.033736 21659 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:00.033797 21659 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:00.033905 21659 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:00.034950 21659 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "d673ca465caf4a679cc7b158b8c085b3"
format_stamp: "Formatted at 2026-04-29 14:10:00 on dist-test-slave-8n77"
server_key: "79aad4caa98ea13f36cfbfab18196947"
server_key_iv: "bb42b48c0d009c98fe5a5c379ca72e84"
server_key_version: "encryptionkey@0"
I20260429 14:10:00.035296 21659 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "d673ca465caf4a679cc7b158b8c085b3"
format_stamp: "Formatted at 2026-04-29 14:10:00 on dist-test-slave-8n77"
server_key: "79aad4caa98ea13f36cfbfab18196947"
server_key_iv: "bb42b48c0d009c98fe5a5c379ca72e84"
server_key_version: "encryptionkey@0"
I20260429 14:10:00.036558 21659 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260429 14:10:00.037526 21674 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:00.037863 21659 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:10:00.037961 21659 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "d673ca465caf4a679cc7b158b8c085b3"
format_stamp: "Formatted at 2026-04-29 14:10:00 on dist-test-slave-8n77"
server_key: "79aad4caa98ea13f36cfbfab18196947"
server_key_iv: "bb42b48c0d009c98fe5a5c379ca72e84"
server_key_version: "encryptionkey@0"
I20260429 14:10:00.038044 21659 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:00.045917 21659 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:00.051909 21659 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:00.052086 21659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:00.057457 21659 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:40915
I20260429 14:10:00.057504 21726 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:40915 every 8 connection(s)
I20260429 14:10:00.057864 21659 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:10:00.058524 21727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:00.061182 21727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d673ca465caf4a679cc7b158b8c085b3: Bootstrap starting.
I20260429 14:10:00.061857 21727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d673ca465caf4a679cc7b158b8c085b3: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:00.062162 21727 log.cc:826] T 00000000000000000000000000000000 P d673ca465caf4a679cc7b158b8c085b3: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:00.062988 21727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d673ca465caf4a679cc7b158b8c085b3: No bootstrap required, opened a new log
I20260429 14:10:00.063673 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 21659
I20260429 14:10:00.063910 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:10:00.064389 2839 external_mini_cluster.cc:1468] Setting key 5380fee083a48b151ce595813233436d
I20260429 14:10:00.065071 21727 raft_consensus.cc:359] T 00000000000000000000000000000000 P d673ca465caf4a679cc7b158b8c085b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d673ca465caf4a679cc7b158b8c085b3" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40915 } }
I20260429 14:10:00.065267 21727 raft_consensus.cc:385] T 00000000000000000000000000000000 P d673ca465caf4a679cc7b158b8c085b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:10:00.065313 21727 raft_consensus.cc:740] T 00000000000000000000000000000000 P d673ca465caf4a679cc7b158b8c085b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d673ca465caf4a679cc7b158b8c085b3, State: Initialized, Role: FOLLOWER
I20260429 14:10:00.065428 21727 consensus_queue.cc:260] T 00000000000000000000000000000000 P d673ca465caf4a679cc7b158b8c085b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d673ca465caf4a679cc7b158b8c085b3" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40915 } }
I20260429 14:10:00.065622 21727 raft_consensus.cc:399] T 00000000000000000000000000000000 P d673ca465caf4a679cc7b158b8c085b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:10:00.065687 21727 raft_consensus.cc:493] T 00000000000000000000000000000000 P d673ca465caf4a679cc7b158b8c085b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:10:00.065740 21727 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d673ca465caf4a679cc7b158b8c085b3 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:10:00.066679 21727 raft_consensus.cc:515] T 00000000000000000000000000000000 P d673ca465caf4a679cc7b158b8c085b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d673ca465caf4a679cc7b158b8c085b3" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40915 } }
I20260429 14:10:00.066812 21727 leader_election.cc:304] T 00000000000000000000000000000000 P d673ca465caf4a679cc7b158b8c085b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d673ca465caf4a679cc7b158b8c085b3; no voters:
I20260429 14:10:00.066982 21727 leader_election.cc:290] T 00000000000000000000000000000000 P d673ca465caf4a679cc7b158b8c085b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:10:00.067051 21732 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d673ca465caf4a679cc7b158b8c085b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:10:00.067277 21727 sys_catalog.cc:565] T 00000000000000000000000000000000 P d673ca465caf4a679cc7b158b8c085b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:10:00.067346 21732 raft_consensus.cc:697] T 00000000000000000000000000000000 P d673ca465caf4a679cc7b158b8c085b3 [term 1 LEADER]: Becoming Leader. State: Replica: d673ca465caf4a679cc7b158b8c085b3, State: Running, Role: LEADER
I20260429 14:10:00.067555 21732 consensus_queue.cc:237] T 00000000000000000000000000000000 P d673ca465caf4a679cc7b158b8c085b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d673ca465caf4a679cc7b158b8c085b3" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40915 } }
I20260429 14:10:00.067898 21732 sys_catalog.cc:455] T 00000000000000000000000000000000 P d673ca465caf4a679cc7b158b8c085b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d673ca465caf4a679cc7b158b8c085b3. Latest consensus state: current_term: 1 leader_uuid: "d673ca465caf4a679cc7b158b8c085b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d673ca465caf4a679cc7b158b8c085b3" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40915 } } }
I20260429 14:10:00.068130 21732 sys_catalog.cc:458] T 00000000000000000000000000000000 P d673ca465caf4a679cc7b158b8c085b3 [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:00.068064 21734 sys_catalog.cc:455] T 00000000000000000000000000000000 P d673ca465caf4a679cc7b158b8c085b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d673ca465caf4a679cc7b158b8c085b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d673ca465caf4a679cc7b158b8c085b3" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40915 } } }
I20260429 14:10:00.068379 21734 sys_catalog.cc:458] T 00000000000000000000000000000000 P d673ca465caf4a679cc7b158b8c085b3 [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:00.068697 21741 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:10:00.069190 21741 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:10:00.070847 21741 catalog_manager.cc:1357] Generated new cluster ID: 87e46762cd0e4511b90cfabe38c9562d
I20260429 14:10:00.070925 21741 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Apr 29 14:10:00 dist-test-slave-8n77 krb5kdc[21643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:10:00.075747 21731 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:00.065057 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54268 (local address 127.2.197.254:40915)
0429 14:10:00.065270 (+ 213us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:00.065280 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:00.065322 (+ 42us) server_negotiation.cc:408] Connection header received
0429 14:10:00.065435 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:00.065442 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:00.065609 (+ 167us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:00.065795 (+ 186us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:00.066843 (+ 1048us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.068281 (+ 1438us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:00.069389 (+ 1108us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.069838 (+ 449us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:00.072018 (+ 2180us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:00.072024 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:00.072024 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:00.072036 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:00.074163 (+ 2127us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:00.074607 (+ 444us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:00.074610 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:00.074610 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:00.074682 (+ 72us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:00.074877 (+ 195us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:00.074878 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:00.074879 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:00.075154 (+ 275us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:00.075297 (+ 143us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:00.075413 (+ 116us) server_negotiation.cc:300] Negotiation successful
0429 14:10:00.075560 (+ 147us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":106,"threads_started":1}
I20260429 14:10:00.081472 21741 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:10:00.082330 21741 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:10:00.088567 21741 catalog_manager.cc:6044] T 00000000000000000000000000000000 P d673ca465caf4a679cc7b158b8c085b3: Generated new TSK 0
I20260429 14:10:00.088747 21741 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:10:00.154807 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.193:0
--local_ip_for_outbound_sockets=127.2.197.193
--webserver_interface=127.2.197.193
--webserver_port=0
--tserver_master_addrs=127.2.197.254:40915
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:35379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:10:00.239391 21755 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:00.239620 21755 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:00.239668 21755 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:10:00.241554 21755 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:00.241670 21755 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:00.241721 21755 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.193
I20260429 14:10:00.243721 21755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:35379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.197.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:40915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.193
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:00.244021 21755 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:00.244226 21755 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:10:00.247483 21761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:00.247529 21763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:00.247496 21760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:00.248011 21755 server_base.cc:1061] running on GCE node
I20260429 14:10:00.248199 21755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:00.248631 21755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:00.249843 21755 hybrid_clock.cc:648] HybridClock initialized: now 1777471800249819 us; error 47 us; skew 500 ppm
Apr 29 14:10:00 dist-test-slave-8n77 krb5kdc[21643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471800, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:10:00.253314 21755 init.cc:377] Logged in from keytab as kudu/127.2.197.193@KRBTEST.COM (short username kudu)
I20260429 14:10:00.253948 21755 webserver.cc:492] Webserver started at http://127.2.197.193:42329/ using document root <none> and password file <none>
I20260429 14:10:00.254161 21755 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:00.254226 21755 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:00.254398 21755 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:00.255419 21755 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-0/data/instance:
uuid: "7c1561d59ab741fb933410a67f83a58c"
format_stamp: "Formatted at 2026-04-29 14:10:00 on dist-test-slave-8n77"
server_key: "afa69b331bb633921b56ed30bc92d87b"
server_key_iv: "8d8ced26bd35cc3314c39e65d3051098"
server_key_version: "encryptionkey@0"
I20260429 14:10:00.255777 21755 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance:
uuid: "7c1561d59ab741fb933410a67f83a58c"
format_stamp: "Formatted at 2026-04-29 14:10:00 on dist-test-slave-8n77"
server_key: "afa69b331bb633921b56ed30bc92d87b"
server_key_iv: "8d8ced26bd35cc3314c39e65d3051098"
server_key_version: "encryptionkey@0"
I20260429 14:10:00.257396 21755 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260429 14:10:00.258543 21770 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:00.258921 21755 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:10:00.259089 21755 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-0/wal
uuid: "7c1561d59ab741fb933410a67f83a58c"
format_stamp: "Formatted at 2026-04-29 14:10:00 on dist-test-slave-8n77"
server_key: "afa69b331bb633921b56ed30bc92d87b"
server_key_iv: "8d8ced26bd35cc3314c39e65d3051098"
server_key_version: "encryptionkey@0"
I20260429 14:10:00.259230 21755 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:00.274238 21755 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:00.277015 21755 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:00.277357 21755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:00.277880 21755 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:10:00.278656 21755 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:10:00.278748 21755 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:00.278789 21755 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:10:00.278822 21755 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:00.286422 21755 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.193:34723
I20260429 14:10:00.286509 21883 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.193:34723 every 8 connection(s)
I20260429 14:10:00.286837 21755 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
I20260429 14:10:00.290252 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 21755
I20260429 14:10:00.290467 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance
I20260429 14:10:00.290670 2839 external_mini_cluster.cc:1468] Setting key 858cb119319c19b8317cc71a96b8f251
Apr 29 14:10:00 dist-test-slave-8n77 krb5kdc[21643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471800, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:10:00.296793 21731 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:00.287726 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:55779 (local address 127.2.197.254:40915)
0429 14:10:00.287833 (+ 107us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:00.287836 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:00.287968 (+ 132us) server_negotiation.cc:408] Connection header received
0429 14:10:00.288479 (+ 511us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:00.288480 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:00.288490 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:00.288566 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:00.289313 (+ 747us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.289846 (+ 533us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:00.290922 (+ 1076us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.291114 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:00.293705 (+ 2591us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:00.293711 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:00.293712 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:00.293727 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:00.295371 (+ 1644us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:00.295878 (+ 507us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:00.295879 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:00.295880 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:00.295911 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:00.296118 (+ 207us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:00.296119 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:00.296120 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:00.296236 (+ 116us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:00.296312 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:00.296576 (+ 264us) server_negotiation.cc:300] Negotiation successful
0429 14:10:00.296694 (+ 118us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260429 14:10:00.296893 21886 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:00.287719 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40915 (local address 127.2.197.193:55779)
0429 14:10:00.287846 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:00.287882 (+ 36us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:00.288402 (+ 520us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:00.288608 (+ 206us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:00.288612 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:00.288910 (+ 298us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:10:00.289169 (+ 259us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:00.289172 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.289957 (+ 785us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:00.289958 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:00.290822 (+ 864us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:00.290826 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.290986 (+ 160us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:00.291982 (+ 996us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:10:00.291992 (+ 10us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:10:00.293590 (+ 1598us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:10:00.295539 (+ 1949us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:00.295548 (+ 9us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:00.295551 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:00.295784 (+ 233us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:00.295978 (+ 194us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:00.295979 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:00.295979 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:00.296051 (+ 72us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:00.296303 (+ 252us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:10:00.296308 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:10:00.296440 (+ 132us) client_negotiation.cc:770] Sending connection context
0429 14:10:00.296558 (+ 118us) client_negotiation.cc:241] Negotiation successful
0429 14:10:00.296706 (+ 148us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":69,"threads_started":1}
I20260429 14:10:00.297456 21884 heartbeater.cc:344] Connected to a master server at 127.2.197.254:40915
I20260429 14:10:00.297571 21884 heartbeater.cc:461] Registering TS with master...
I20260429 14:10:00.297811 21884 heartbeater.cc:507] Master 127.2.197.254:40915 requested a full tablet report, sending...
I20260429 14:10:00.298475 21691 ts_manager.cc:194] Registered new tserver with Master: 7c1561d59ab741fb933410a67f83a58c (127.2.197.193:34723)
WARNING: no policy specified for kudu/127.2.197.194@KRBTEST.COM; defaulting to no policy
I20260429 14:10:00.299613 21691 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.193@KRBTEST.COM'} at 127.2.197.193:55779
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:10:00.347421 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.194:0
--local_ip_for_outbound_sockets=127.2.197.194
--webserver_interface=127.2.197.194
--webserver_port=0
--tserver_master_addrs=127.2.197.254:40915
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:35379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:10:00.432238 21891 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:00.432484 21891 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:00.432523 21891 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:10:00.434116 21891 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:00.434214 21891 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:00.434266 21891 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.194
I20260429 14:10:00.436067 21891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:35379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.197.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:40915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.194
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:00.436372 21891 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:00.436578 21891 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260429 14:10:00.439486 21891 server_base.cc:1061] running on GCE node
W20260429 14:10:00.439487 21899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:00.439709 21897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:00.439751 21896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:00.439951 21891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:00.440225 21891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:00.441426 21891 hybrid_clock.cc:648] HybridClock initialized: now 1777471800441423 us; error 45 us; skew 500 ppm
Apr 29 14:10:00 dist-test-slave-8n77 krb5kdc[21643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471800, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:10:00.444021 21891 init.cc:377] Logged in from keytab as kudu/127.2.197.194@KRBTEST.COM (short username kudu)
I20260429 14:10:00.444629 21891 webserver.cc:492] Webserver started at http://127.2.197.194:45989/ using document root <none> and password file <none>
I20260429 14:10:00.444787 21891 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:00.444823 21891 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:00.444898 21891 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:00.445864 21891 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-1/data/instance:
uuid: "81f9542f59404c25b26b1c0556af6fc7"
format_stamp: "Formatted at 2026-04-29 14:10:00 on dist-test-slave-8n77"
server_key: "4edd5eda1084d154be4f6d7ad9efa7d8"
server_key_iv: "807b8ee5b0f49b0acfa564c9e9ca672f"
server_key_version: "encryptionkey@0"
I20260429 14:10:00.446157 21891 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance:
uuid: "81f9542f59404c25b26b1c0556af6fc7"
format_stamp: "Formatted at 2026-04-29 14:10:00 on dist-test-slave-8n77"
server_key: "4edd5eda1084d154be4f6d7ad9efa7d8"
server_key_iv: "807b8ee5b0f49b0acfa564c9e9ca672f"
server_key_version: "encryptionkey@0"
I20260429 14:10:00.447544 21891 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:10:00.448529 21906 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:00.448755 21891 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:10:00.448828 21891 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-1/wal
uuid: "81f9542f59404c25b26b1c0556af6fc7"
format_stamp: "Formatted at 2026-04-29 14:10:00 on dist-test-slave-8n77"
server_key: "4edd5eda1084d154be4f6d7ad9efa7d8"
server_key_iv: "807b8ee5b0f49b0acfa564c9e9ca672f"
server_key_version: "encryptionkey@0"
I20260429 14:10:00.448882 21891 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:00.462081 21891 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:00.465126 21891 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:00.465270 21891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:00.465487 21891 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:10:00.465833 21891 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:10:00.465868 21891 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:00.465893 21891 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:10:00.465905 21891 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:00.474154 21891 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.194:38155
I20260429 14:10:00.474246 22019 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.194:38155 every 8 connection(s)
I20260429 14:10:00.474584 21891 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
Apr 29 14:10:00 dist-test-slave-8n77 krb5kdc[21643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471800, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:10:00.483124 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 21891
I20260429 14:10:00.483285 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance
I20260429 14:10:00.483518 2839 external_mini_cluster.cc:1468] Setting key 64f774f03aaefb7e94654750f3c58df2
I20260429 14:10:00.486148 21731 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:00.475424 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:55151 (local address 127.2.197.254:40915)
0429 14:10:00.475545 (+ 121us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:00.475548 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:00.475713 (+ 165us) server_negotiation.cc:408] Connection header received
0429 14:10:00.476286 (+ 573us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:00.476288 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:00.476307 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:00.476382 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:00.477121 (+ 739us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.477659 (+ 538us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:00.478470 (+ 811us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.478685 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:00.482828 (+ 4143us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:00.482835 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:00.482836 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:00.482854 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:00.484691 (+ 1837us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:00.485159 (+ 468us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:00.485161 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:00.485161 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:00.485195 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:00.485435 (+ 240us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:00.485436 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:00.485436 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:00.485564 (+ 128us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:00.485651 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:00.485943 (+ 292us) server_negotiation.cc:300] Negotiation successful
0429 14:10:00.486050 (+ 107us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260429 14:10:00.486343 22022 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:00.475320 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40915 (local address 127.2.197.194:55151)
0429 14:10:00.475595 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:00.475624 (+ 29us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:00.476131 (+ 507us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:00.476448 (+ 317us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:00.476452 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:00.476763 (+ 311us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:10:00.477013 (+ 250us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:00.477016 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.477911 (+ 895us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:00.477914 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:00.478371 (+ 457us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:00.478373 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.478524 (+ 151us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:00.479941 (+ 1417us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:10:00.479952 (+ 11us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:10:00.482690 (+ 2738us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:10:00.484793 (+ 2103us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:00.484802 (+ 9us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:00.484806 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:00.485058 (+ 252us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:00.485269 (+ 211us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:00.485270 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:00.485271 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:00.485351 (+ 80us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:00.485640 (+ 289us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:10:00.485646 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:10:00.485793 (+ 147us) client_negotiation.cc:770] Sending connection context
0429 14:10:00.485923 (+ 130us) client_negotiation.cc:241] Negotiation successful
0429 14:10:00.486082 (+ 159us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":108,"threads_started":1}
I20260429 14:10:00.487550 22020 heartbeater.cc:344] Connected to a master server at 127.2.197.254:40915
I20260429 14:10:00.487715 22020 heartbeater.cc:461] Registering TS with master...
I20260429 14:10:00.488004 22020 heartbeater.cc:507] Master 127.2.197.254:40915 requested a full tablet report, sending...
I20260429 14:10:00.488626 21691 ts_manager.cc:194] Registered new tserver with Master: 81f9542f59404c25b26b1c0556af6fc7 (127.2.197.194:38155)
I20260429 14:10:00.489061 21691 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.194@KRBTEST.COM'} at 127.2.197.194:55151
WARNING: no policy specified for kudu/127.2.197.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:10:00.539449 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.195:0
--local_ip_for_outbound_sockets=127.2.197.195
--webserver_interface=127.2.197.195
--webserver_port=0
--tserver_master_addrs=127.2.197.254:40915
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:35379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:10:00.624833 22027 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:00.625041 22027 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:00.625067 22027 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:10:00.627210 22027 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:00.627295 22027 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:00.627341 22027 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.195
I20260429 14:10:00.629308 22027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:35379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.197.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:40915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.195
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:00.629580 22027 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:00.629837 22027 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:10:00.633018 22035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:00.632999 22033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:00.632992 22032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:00.633309 22027 server_base.cc:1061] running on GCE node
I20260429 14:10:00.633800 22027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:00.634066 22027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:00.635305 22027 hybrid_clock.cc:648] HybridClock initialized: now 1777471800635203 us; error 90 us; skew 500 ppm
Apr 29 14:10:00 dist-test-slave-8n77 krb5kdc[21643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471800, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:10:00.638698 22027 init.cc:377] Logged in from keytab as kudu/127.2.197.195@KRBTEST.COM (short username kudu)
I20260429 14:10:00.639319 22027 webserver.cc:492] Webserver started at http://127.2.197.195:44903/ using document root <none> and password file <none>
I20260429 14:10:00.639539 22027 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:00.639612 22027 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:00.639712 22027 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:00.640702 22027 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-2/data/instance:
uuid: "c5c05b84f90f4495870099021214efcd"
format_stamp: "Formatted at 2026-04-29 14:10:00 on dist-test-slave-8n77"
server_key: "b5c55be50665a6dc63ca3d964c89a9a0"
server_key_iv: "c3a95bb12df5b1789b8c5efedcfbfd64"
server_key_version: "encryptionkey@0"
I20260429 14:10:00.641048 22027 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance:
uuid: "c5c05b84f90f4495870099021214efcd"
format_stamp: "Formatted at 2026-04-29 14:10:00 on dist-test-slave-8n77"
server_key: "b5c55be50665a6dc63ca3d964c89a9a0"
server_key_iv: "c3a95bb12df5b1789b8c5efedcfbfd64"
server_key_version: "encryptionkey@0"
I20260429 14:10:00.642550 22027 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260429 14:10:00.643420 22042 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:00.643616 22027 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:10:00.643699 22027 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-2/wal
uuid: "c5c05b84f90f4495870099021214efcd"
format_stamp: "Formatted at 2026-04-29 14:10:00 on dist-test-slave-8n77"
server_key: "b5c55be50665a6dc63ca3d964c89a9a0"
server_key_iv: "c3a95bb12df5b1789b8c5efedcfbfd64"
server_key_version: "encryptionkey@0"
I20260429 14:10:00.643781 22027 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:00.655035 22027 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:00.657207 22027 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:00.657378 22027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:00.657617 22027 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:10:00.657934 22027 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:10:00.657989 22027 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:00.658033 22027 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:10:00.658061 22027 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:00.664739 22027 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.195:36471
I20260429 14:10:00.664781 22155 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.195:36471 every 8 connection(s)
I20260429 14:10:00.665112 22027 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
Apr 29 14:10:00 dist-test-slave-8n77 krb5kdc[21643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471800, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:10:00.674942 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 22027
I20260429 14:10:00.675084 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance
I20260429 14:10:00.675282 2839 external_mini_cluster.cc:1468] Setting key 9fef71cf2c4f8cf649e017bc66a3838a
I20260429 14:10:00.675868 21731 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:00.665895 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:57385 (local address 127.2.197.254:40915)
0429 14:10:00.665968 (+ 73us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:00.665970 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:00.666185 (+ 215us) server_negotiation.cc:408] Connection header received
0429 14:10:00.666817 (+ 632us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:00.666818 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:00.666832 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:00.666899 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:00.667681 (+ 782us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.668251 (+ 570us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:00.668891 (+ 640us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.669064 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:00.671905 (+ 2841us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:00.671911 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:00.671912 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:00.671921 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:00.673603 (+ 1682us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:00.674063 (+ 460us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:00.674064 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:00.674064 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:00.674089 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:00.674623 (+ 534us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:00.674624 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:00.674625 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:00.674733 (+ 108us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:00.674861 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:00.675639 (+ 778us) server_negotiation.cc:300] Negotiation successful
0429 14:10:00.675756 (+ 117us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260429 14:10:00.675981 22158 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:00.665916 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40915 (local address 127.2.197.195:57385)
0429 14:10:00.666074 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:00.666103 (+ 29us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:00.666701 (+ 598us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:00.666958 (+ 257us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:00.666966 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:00.667308 (+ 342us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:10:00.667567 (+ 259us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:00.667570 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.668394 (+ 824us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:00.668395 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:00.668773 (+ 378us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:00.668774 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.668945 (+ 171us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:00.669935 (+ 990us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:10:00.669944 (+ 9us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:10:00.671793 (+ 1849us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:10:00.673712 (+ 1919us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:00.673726 (+ 14us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:00.673728 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:00.673970 (+ 242us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:00.674245 (+ 275us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:00.674247 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:00.674247 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:00.674518 (+ 271us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:00.674852 (+ 334us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:10:00.674858 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:10:00.675321 (+ 463us) client_negotiation.cc:770] Sending connection context
0429 14:10:00.675549 (+ 228us) client_negotiation.cc:241] Negotiation successful
0429 14:10:00.675715 (+ 166us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
I20260429 14:10:00.676520 22156 heartbeater.cc:344] Connected to a master server at 127.2.197.254:40915
I20260429 14:10:00.676617 22156 heartbeater.cc:461] Registering TS with master...
I20260429 14:10:00.676820 22156 heartbeater.cc:507] Master 127.2.197.254:40915 requested a full tablet report, sending...
I20260429 14:10:00.677388 21691 ts_manager.cc:194] Registered new tserver with Master: c5c05b84f90f4495870099021214efcd (127.2.197.195:36471)
I20260429 14:10:00.677912 21691 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.195@KRBTEST.COM'} at 127.2.197.195:57385
I20260429 14:10:00.688159 2839 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260429 14:10:00.696492 21731 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:00.689731 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54278 (local address 127.2.197.254:40915)
0429 14:10:00.689856 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:00.689859 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:00.689876 (+ 17us) server_negotiation.cc:408] Connection header received
0429 14:10:00.689934 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:00.689935 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:00.689947 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:00.690012 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:00.690886 (+ 874us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.691419 (+ 533us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:00.692043 (+ 624us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.692182 (+ 139us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:00.693474 (+ 1292us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:00.693480 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:00.693481 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:00.693490 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:00.695012 (+ 1522us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:00.695442 (+ 430us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:00.695443 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:00.695444 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:00.695463 (+ 19us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:00.695668 (+ 205us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:00.695668 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:00.695669 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:00.695776 (+ 107us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:00.695868 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:00.696361 (+ 493us) server_negotiation.cc:300] Negotiation successful
0429 14:10:00.696420 (+ 59us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260429 14:10:00.698616 21691 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54278:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 14:10:00.700011 21691 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 14:10:00.706166 22171 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:00.703172 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:38155 (local address 127.0.0.1:48422)
0429 14:10:00.703271 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:00.703284 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:00.703352 (+ 68us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:00.703754 (+ 402us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:00.703755 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:00.703758 (+ 3us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:10:00.703930 (+ 172us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:00.703931 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.704994 (+ 1063us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:00.704995 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:00.705826 (+ 831us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:00.705828 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.705912 (+ 84us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:00.705959 (+ 47us) client_negotiation.cc:770] Sending connection context
0429 14:10:00.706031 (+ 72us) client_negotiation.cc:241] Negotiation successful
0429 14:10:00.706078 (+ 47us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":60,"threads_started":1}
I20260429 14:10:00.706574 22169 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:00.702995 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:36471 (local address 127.0.0.1:41836)
0429 14:10:00.703115 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:00.703125 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:00.703177 (+ 52us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:00.703711 (+ 534us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:00.703712 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:00.703715 (+ 3us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:10:00.704073 (+ 358us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:00.704076 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.705415 (+ 1339us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:00.705416 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:00.706363 (+ 947us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:00.706367 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.706435 (+ 68us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:00.706441 (+ 6us) client_negotiation.cc:770] Sending connection context
0429 14:10:00.706477 (+ 36us) client_negotiation.cc:241] Negotiation successful
0429 14:10:00.706516 (+ 39us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":84,"threads_started":1}
I20260429 14:10:00.706681 22167 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:00.702773 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:34723 (local address 127.0.0.1:51820)
0429 14:10:00.702936 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:00.702971 (+ 35us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:00.703062 (+ 91us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:00.703420 (+ 358us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:00.703423 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:00.703426 (+ 3us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:10:00.703736 (+ 310us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:00.703739 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.705030 (+ 1291us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:00.705031 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:00.706263 (+ 1232us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:00.706306 (+ 43us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.706391 (+ 85us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:00.706398 (+ 7us) client_negotiation.cc:770] Sending connection context
0429 14:10:00.706432 (+ 34us) client_negotiation.cc:241] Negotiation successful
0429 14:10:00.706499 (+ 67us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":73,"threads_started":1}
I20260429 14:10:00.707450 22170 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:00.703098 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41836 (local address 127.2.197.195:36471)
0429 14:10:00.703542 (+ 444us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:00.703549 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:00.703558 (+ 9us) server_negotiation.cc:408] Connection header received
0429 14:10:00.703592 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:00.703594 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:00.703610 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:00.703691 (+ 81us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:10:00.704153 (+ 462us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.705327 (+ 1174us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:00.706840 (+ 1513us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.707244 (+ 404us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:00.707289 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:00.707354 (+ 65us) server_negotiation.cc:300] Negotiation successful
0429 14:10:00.707385 (+ 31us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":80,"threads_started":1}
I20260429 14:10:00.708026 22090 tablet_service.cc:1511] Processing CreateTablet for tablet 4876c507b5e4457ab4693f674db59207 (DEFAULT_TABLE table=test-table [id=b4d1730a63e34710b4305a986f8648b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:10:00.708091 22172 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:00.703402 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48422 (local address 127.2.197.194:38155)
0429 14:10:00.703588 (+ 186us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:00.703595 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:00.703604 (+ 9us) server_negotiation.cc:408] Connection header received
0429 14:10:00.703641 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:00.703643 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:00.703659 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:00.703985 (+ 326us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:10:00.704039 (+ 54us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.704896 (+ 857us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:00.706718 (+ 1822us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.707788 (+ 1070us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:00.707881 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:00.707922 (+ 41us) server_negotiation.cc:300] Negotiation successful
0429 14:10:00.707982 (+ 60us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":113,"threads_started":1}
I20260429 14:10:00.708391 22090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4876c507b5e4457ab4693f674db59207. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:00.708406 22168 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:00.702926 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51820 (local address 127.2.197.193:34723)
0429 14:10:00.703190 (+ 264us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:00.703216 (+ 26us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:00.703225 (+ 9us) server_negotiation.cc:408] Connection header received
0429 14:10:00.703298 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:00.703299 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:00.703312 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:00.703400 (+ 88us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:10:00.703798 (+ 398us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.704934 (+ 1136us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:00.707573 (+ 2639us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.708155 (+ 582us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:00.708198 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:00.708233 (+ 35us) server_negotiation.cc:300] Negotiation successful
0429 14:10:00.708298 (+ 65us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":102,"threads_started":1}
I20260429 14:10:00.708877 21954 tablet_service.cc:1511] Processing CreateTablet for tablet 4876c507b5e4457ab4693f674db59207 (DEFAULT_TABLE table=test-table [id=b4d1730a63e34710b4305a986f8648b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:10:00.709000 21818 tablet_service.cc:1511] Processing CreateTablet for tablet 4876c507b5e4457ab4693f674db59207 (DEFAULT_TABLE table=test-table [id=b4d1730a63e34710b4305a986f8648b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:10:00.709160 21954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4876c507b5e4457ab4693f674db59207. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:00.709286 21818 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4876c507b5e4457ab4693f674db59207. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:00.711618 22173 tablet_bootstrap.cc:492] T 4876c507b5e4457ab4693f674db59207 P c5c05b84f90f4495870099021214efcd: Bootstrap starting.
I20260429 14:10:00.712409 22173 tablet_bootstrap.cc:654] T 4876c507b5e4457ab4693f674db59207 P c5c05b84f90f4495870099021214efcd: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:00.712327 22174 tablet_bootstrap.cc:492] T 4876c507b5e4457ab4693f674db59207 P 81f9542f59404c25b26b1c0556af6fc7: Bootstrap starting.
I20260429 14:10:00.712889 22175 tablet_bootstrap.cc:492] T 4876c507b5e4457ab4693f674db59207 P 7c1561d59ab741fb933410a67f83a58c: Bootstrap starting.
I20260429 14:10:00.713032 22174 tablet_bootstrap.cc:654] T 4876c507b5e4457ab4693f674db59207 P 81f9542f59404c25b26b1c0556af6fc7: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:00.712833 22173 log.cc:826] T 4876c507b5e4457ab4693f674db59207 P c5c05b84f90f4495870099021214efcd: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:00.713366 22174 log.cc:826] T 4876c507b5e4457ab4693f674db59207 P 81f9542f59404c25b26b1c0556af6fc7: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:00.713634 22175 tablet_bootstrap.cc:654] T 4876c507b5e4457ab4693f674db59207 P 7c1561d59ab741fb933410a67f83a58c: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:00.713815 22173 tablet_bootstrap.cc:492] T 4876c507b5e4457ab4693f674db59207 P c5c05b84f90f4495870099021214efcd: No bootstrap required, opened a new log
I20260429 14:10:00.713883 22173 ts_tablet_manager.cc:1403] T 4876c507b5e4457ab4693f674db59207 P c5c05b84f90f4495870099021214efcd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260429 14:10:00.713958 22175 log.cc:826] T 4876c507b5e4457ab4693f674db59207 P 7c1561d59ab741fb933410a67f83a58c: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:00.714077 22174 tablet_bootstrap.cc:492] T 4876c507b5e4457ab4693f674db59207 P 81f9542f59404c25b26b1c0556af6fc7: No bootstrap required, opened a new log
I20260429 14:10:00.714134 22174 ts_tablet_manager.cc:1403] T 4876c507b5e4457ab4693f674db59207 P 81f9542f59404c25b26b1c0556af6fc7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260429 14:10:00.714773 22175 tablet_bootstrap.cc:492] T 4876c507b5e4457ab4693f674db59207 P 7c1561d59ab741fb933410a67f83a58c: No bootstrap required, opened a new log
I20260429 14:10:00.714852 22175 ts_tablet_manager.cc:1403] T 4876c507b5e4457ab4693f674db59207 P 7c1561d59ab741fb933410a67f83a58c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260429 14:10:00.715638 22174 raft_consensus.cc:359] T 4876c507b5e4457ab4693f674db59207 P 81f9542f59404c25b26b1c0556af6fc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1561d59ab741fb933410a67f83a58c" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34723 } } peers { permanent_uuid: "c5c05b84f90f4495870099021214efcd" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36471 } } peers { permanent_uuid: "81f9542f59404c25b26b1c0556af6fc7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38155 } }
I20260429 14:10:00.715813 22174 raft_consensus.cc:385] T 4876c507b5e4457ab4693f674db59207 P 81f9542f59404c25b26b1c0556af6fc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:10:00.715715 22173 raft_consensus.cc:359] T 4876c507b5e4457ab4693f674db59207 P c5c05b84f90f4495870099021214efcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1561d59ab741fb933410a67f83a58c" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34723 } } peers { permanent_uuid: "c5c05b84f90f4495870099021214efcd" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36471 } } peers { permanent_uuid: "81f9542f59404c25b26b1c0556af6fc7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38155 } }
I20260429 14:10:00.715857 22174 raft_consensus.cc:740] T 4876c507b5e4457ab4693f674db59207 P 81f9542f59404c25b26b1c0556af6fc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81f9542f59404c25b26b1c0556af6fc7, State: Initialized, Role: FOLLOWER
I20260429 14:10:00.715859 22173 raft_consensus.cc:385] T 4876c507b5e4457ab4693f674db59207 P c5c05b84f90f4495870099021214efcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:10:00.715889 22173 raft_consensus.cc:740] T 4876c507b5e4457ab4693f674db59207 P c5c05b84f90f4495870099021214efcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5c05b84f90f4495870099021214efcd, State: Initialized, Role: FOLLOWER
I20260429 14:10:00.715951 22174 consensus_queue.cc:260] T 4876c507b5e4457ab4693f674db59207 P 81f9542f59404c25b26b1c0556af6fc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1561d59ab741fb933410a67f83a58c" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34723 } } peers { permanent_uuid: "c5c05b84f90f4495870099021214efcd" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36471 } } peers { permanent_uuid: "81f9542f59404c25b26b1c0556af6fc7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38155 } }
I20260429 14:10:00.716005 22173 consensus_queue.cc:260] T 4876c507b5e4457ab4693f674db59207 P c5c05b84f90f4495870099021214efcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1561d59ab741fb933410a67f83a58c" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34723 } } peers { permanent_uuid: "c5c05b84f90f4495870099021214efcd" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36471 } } peers { permanent_uuid: "81f9542f59404c25b26b1c0556af6fc7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38155 } }
I20260429 14:10:00.716279 22174 ts_tablet_manager.cc:1434] T 4876c507b5e4457ab4693f674db59207 P 81f9542f59404c25b26b1c0556af6fc7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260429 14:10:00.716298 22020 heartbeater.cc:499] Master 127.2.197.254:40915 was elected leader, sending a full tablet report...
I20260429 14:10:00.716498 22173 ts_tablet_manager.cc:1434] T 4876c507b5e4457ab4693f674db59207 P c5c05b84f90f4495870099021214efcd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260429 14:10:00.716661 22156 heartbeater.cc:499] Master 127.2.197.254:40915 was elected leader, sending a full tablet report...
I20260429 14:10:00.716770 22175 raft_consensus.cc:359] T 4876c507b5e4457ab4693f674db59207 P 7c1561d59ab741fb933410a67f83a58c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1561d59ab741fb933410a67f83a58c" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34723 } } peers { permanent_uuid: "c5c05b84f90f4495870099021214efcd" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36471 } } peers { permanent_uuid: "81f9542f59404c25b26b1c0556af6fc7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38155 } }
I20260429 14:10:00.716928 22175 raft_consensus.cc:385] T 4876c507b5e4457ab4693f674db59207 P 7c1561d59ab741fb933410a67f83a58c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:10:00.716958 22175 raft_consensus.cc:740] T 4876c507b5e4457ab4693f674db59207 P 7c1561d59ab741fb933410a67f83a58c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c1561d59ab741fb933410a67f83a58c, State: Initialized, Role: FOLLOWER
I20260429 14:10:00.717041 22175 consensus_queue.cc:260] T 4876c507b5e4457ab4693f674db59207 P 7c1561d59ab741fb933410a67f83a58c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1561d59ab741fb933410a67f83a58c" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34723 } } peers { permanent_uuid: "c5c05b84f90f4495870099021214efcd" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36471 } } peers { permanent_uuid: "81f9542f59404c25b26b1c0556af6fc7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38155 } }
I20260429 14:10:00.717288 22175 ts_tablet_manager.cc:1434] T 4876c507b5e4457ab4693f674db59207 P 7c1561d59ab741fb933410a67f83a58c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260429 14:10:00.717327 21884 heartbeater.cc:499] Master 127.2.197.254:40915 was elected leader, sending a full tablet report...
W20260429 14:10:00.725152 22021 tablet.cc:2404] T 4876c507b5e4457ab4693f674db59207 P 81f9542f59404c25b26b1c0556af6fc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 14:10:00.734715 22181 raft_consensus.cc:493] T 4876c507b5e4457ab4693f674db59207 P 7c1561d59ab741fb933410a67f83a58c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:10:00.734845 22181 raft_consensus.cc:515] T 4876c507b5e4457ab4693f674db59207 P 7c1561d59ab741fb933410a67f83a58c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1561d59ab741fb933410a67f83a58c" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34723 } } peers { permanent_uuid: "c5c05b84f90f4495870099021214efcd" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36471 } } peers { permanent_uuid: "81f9542f59404c25b26b1c0556af6fc7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38155 } }
I20260429 14:10:00.735247 22181 leader_election.cc:290] T 4876c507b5e4457ab4693f674db59207 P 7c1561d59ab741fb933410a67f83a58c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5c05b84f90f4495870099021214efcd (127.2.197.195:36471), 81f9542f59404c25b26b1c0556af6fc7 (127.2.197.194:38155)
I20260429 14:10:00.738818 21886 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:00.735746 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:36471 (local address 127.2.197.193:46505)
0429 14:10:00.735845 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:00.735861 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:00.735941 (+ 80us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:00.736293 (+ 352us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:00.736295 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:00.736299 (+ 4us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:10:00.736590 (+ 291us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:00.736592 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.737615 (+ 1023us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:00.737616 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:00.738546 (+ 930us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:00.738549 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.738630 (+ 81us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:00.738636 (+ 6us) client_negotiation.cc:770] Sending connection context
0429 14:10:00.738675 (+ 39us) client_negotiation.cc:241] Negotiation successful
0429 14:10:00.738716 (+ 41us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260429 14:10:00.738827 22182 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:00.735973 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:38155 (local address 127.2.197.193:49667)
0429 14:10:00.736349 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:00.736365 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:00.736474 (+ 109us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:00.736637 (+ 163us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:00.736638 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:00.736640 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:10:00.736864 (+ 224us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:00.736866 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.737544 (+ 678us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:00.737545 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:00.738548 (+ 1003us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:00.738549 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.738649 (+ 100us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:00.738652 (+ 3us) client_negotiation.cc:770] Sending connection context
0429 14:10:00.738689 (+ 37us) client_negotiation.cc:241] Negotiation successful
0429 14:10:00.738716 (+ 27us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":162,"threads_started":1}
I20260429 14:10:00.739250 22172 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:00.736077 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:49667 (local address 127.2.197.194:38155)
0429 14:10:00.736125 (+ 48us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:00.736127 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:00.736485 (+ 358us) server_negotiation.cc:408] Connection header received
0429 14:10:00.736545 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:00.736547 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:00.736557 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:00.736614 (+ 57us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:10:00.736961 (+ 347us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.737458 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:00.738658 (+ 1200us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.739103 (+ 445us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:00.739112 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:00.739135 (+ 23us) server_negotiation.cc:300] Negotiation successful
0429 14:10:00.739166 (+ 31us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260429 14:10:00.739574 21974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4876c507b5e4457ab4693f674db59207" candidate_uuid: "7c1561d59ab741fb933410a67f83a58c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81f9542f59404c25b26b1c0556af6fc7" is_pre_election: true
I20260429 14:10:00.739631 22170 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:00.735806 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:46505 (local address 127.2.197.195:36471)
0429 14:10:00.735871 (+ 65us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:00.735873 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:00.735923 (+ 50us) server_negotiation.cc:408] Connection header received
0429 14:10:00.736051 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:00.736054 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:00.736070 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:00.736687 (+ 617us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:10:00.736746 (+ 59us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.737534 (+ 788us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:00.738960 (+ 1426us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.739505 (+ 545us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:00.739513 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:00.739537 (+ 24us) server_negotiation.cc:300] Negotiation successful
0429 14:10:00.739574 (+ 37us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260429 14:10:00.739753 21974 raft_consensus.cc:2468] T 4876c507b5e4457ab4693f674db59207 P 81f9542f59404c25b26b1c0556af6fc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c1561d59ab741fb933410a67f83a58c in term 0.
I20260429 14:10:00.739975 22110 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4876c507b5e4457ab4693f674db59207" candidate_uuid: "7c1561d59ab741fb933410a67f83a58c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5c05b84f90f4495870099021214efcd" is_pre_election: true
I20260429 14:10:00.740028 21771 leader_election.cc:304] T 4876c507b5e4457ab4693f674db59207 P 7c1561d59ab741fb933410a67f83a58c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c1561d59ab741fb933410a67f83a58c, 81f9542f59404c25b26b1c0556af6fc7; no voters:
I20260429 14:10:00.740111 22110 raft_consensus.cc:2468] T 4876c507b5e4457ab4693f674db59207 P c5c05b84f90f4495870099021214efcd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c1561d59ab741fb933410a67f83a58c in term 0.
I20260429 14:10:00.740399 22181 raft_consensus.cc:2804] T 4876c507b5e4457ab4693f674db59207 P 7c1561d59ab741fb933410a67f83a58c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 14:10:00.740500 22181 raft_consensus.cc:493] T 4876c507b5e4457ab4693f674db59207 P 7c1561d59ab741fb933410a67f83a58c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 14:10:00.740532 22181 raft_consensus.cc:3060] T 4876c507b5e4457ab4693f674db59207 P 7c1561d59ab741fb933410a67f83a58c [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:10:00.741258 22181 raft_consensus.cc:515] T 4876c507b5e4457ab4693f674db59207 P 7c1561d59ab741fb933410a67f83a58c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1561d59ab741fb933410a67f83a58c" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34723 } } peers { permanent_uuid: "c5c05b84f90f4495870099021214efcd" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36471 } } peers { permanent_uuid: "81f9542f59404c25b26b1c0556af6fc7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38155 } }
I20260429 14:10:00.741427 22181 leader_election.cc:290] T 4876c507b5e4457ab4693f674db59207 P 7c1561d59ab741fb933410a67f83a58c [CANDIDATE]: Term 1 election: Requested vote from peers c5c05b84f90f4495870099021214efcd (127.2.197.195:36471), 81f9542f59404c25b26b1c0556af6fc7 (127.2.197.194:38155)
I20260429 14:10:00.741715 22110 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4876c507b5e4457ab4693f674db59207" candidate_uuid: "7c1561d59ab741fb933410a67f83a58c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5c05b84f90f4495870099021214efcd"
I20260429 14:10:00.741739 21974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4876c507b5e4457ab4693f674db59207" candidate_uuid: "7c1561d59ab741fb933410a67f83a58c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81f9542f59404c25b26b1c0556af6fc7"
I20260429 14:10:00.741812 22110 raft_consensus.cc:3060] T 4876c507b5e4457ab4693f674db59207 P c5c05b84f90f4495870099021214efcd [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:10:00.741812 21974 raft_consensus.cc:3060] T 4876c507b5e4457ab4693f674db59207 P 81f9542f59404c25b26b1c0556af6fc7 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:10:00.742973 21974 raft_consensus.cc:2468] T 4876c507b5e4457ab4693f674db59207 P 81f9542f59404c25b26b1c0556af6fc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c1561d59ab741fb933410a67f83a58c in term 1.
I20260429 14:10:00.742973 22110 raft_consensus.cc:2468] T 4876c507b5e4457ab4693f674db59207 P c5c05b84f90f4495870099021214efcd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c1561d59ab741fb933410a67f83a58c in term 1.
I20260429 14:10:00.743203 21771 leader_election.cc:304] T 4876c507b5e4457ab4693f674db59207 P 7c1561d59ab741fb933410a67f83a58c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c1561d59ab741fb933410a67f83a58c, c5c05b84f90f4495870099021214efcd; no voters:
I20260429 14:10:00.743342 22181 raft_consensus.cc:2804] T 4876c507b5e4457ab4693f674db59207 P 7c1561d59ab741fb933410a67f83a58c [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:10:00.743553 22181 raft_consensus.cc:697] T 4876c507b5e4457ab4693f674db59207 P 7c1561d59ab741fb933410a67f83a58c [term 1 LEADER]: Becoming Leader. State: Replica: 7c1561d59ab741fb933410a67f83a58c, State: Running, Role: LEADER
I20260429 14:10:00.743671 22181 consensus_queue.cc:237] T 4876c507b5e4457ab4693f674db59207 P 7c1561d59ab741fb933410a67f83a58c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1561d59ab741fb933410a67f83a58c" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34723 } } peers { permanent_uuid: "c5c05b84f90f4495870099021214efcd" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36471 } } peers { permanent_uuid: "81f9542f59404c25b26b1c0556af6fc7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38155 } }
I20260429 14:10:00.744755 21691 catalog_manager.cc:5671] T 4876c507b5e4457ab4693f674db59207 P 7c1561d59ab741fb933410a67f83a58c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c1561d59ab741fb933410a67f83a58c (127.2.197.193). New cstate: current_term: 1 leader_uuid: "7c1561d59ab741fb933410a67f83a58c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1561d59ab741fb933410a67f83a58c" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 34723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5c05b84f90f4495870099021214efcd" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81f9542f59404c25b26b1c0556af6fc7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38155 } health_report { overall_health: UNKNOWN } } }
I20260429 14:10:00.758905 22168 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:00.756523 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51826 (local address 127.2.197.193:34723)
0429 14:10:00.756596 (+ 73us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:00.756598 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:00.756609 (+ 11us) server_negotiation.cc:408] Connection header received
0429 14:10:00.756672 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:00.756673 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:00.756688 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:00.756747 (+ 59us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:10:00.757004 (+ 257us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.757515 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:00.758175 (+ 660us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.758419 (+ 244us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:00.758590 (+ 171us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:10:00.758673 (+ 83us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:10:00.758736 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:00.758835 (+ 99us) server_negotiation.cc:300] Negotiation successful
0429 14:10:00.758844 (+ 9us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260429 14:10:00.760983 21974 raft_consensus.cc:1275] T 4876c507b5e4457ab4693f674db59207 P 81f9542f59404c25b26b1c0556af6fc7 [term 1 FOLLOWER]: Refusing update from remote peer 7c1561d59ab741fb933410a67f83a58c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:10:00.761018 22110 raft_consensus.cc:1275] T 4876c507b5e4457ab4693f674db59207 P c5c05b84f90f4495870099021214efcd [term 1 FOLLOWER]: Refusing update from remote peer 7c1561d59ab741fb933410a67f83a58c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:10:00.761339 22181 consensus_queue.cc:1048] T 4876c507b5e4457ab4693f674db59207 P 7c1561d59ab741fb933410a67f83a58c [LEADER]: Connected to new peer: Peer: permanent_uuid: "81f9542f59404c25b26b1c0556af6fc7" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:10:00.761507 22183 consensus_queue.cc:1048] T 4876c507b5e4457ab4693f674db59207 P 7c1561d59ab741fb933410a67f83a58c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c05b84f90f4495870099021214efcd" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 36471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:10:00.764267 22187 mvcc.cc:204] Tried to move back new op lower bound from 7280524495914201088 to 7280524495846723584. Current Snapshot: MvccSnapshot[applied={T|T < 7280524495914201088}]
I20260429 14:10:00.764492 22188 mvcc.cc:204] Tried to move back new op lower bound from 7280524495914201088 to 7280524495846723584. Current Snapshot: MvccSnapshot[applied={T|T < 7280524495914201088}]
I20260429 14:10:00.766571 21690 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:10:00.766677 21690 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:10:00.767503 21690 catalog_manager.cc:5958] T 00000000000000000000000000000000 P d673ca465caf4a679cc7b158b8c085b3: Sending DeleteTablet for 3 replicas of tablet 4876c507b5e4457ab4693f674db59207
I20260429 14:10:00.767809 21818 tablet_service.cc:1558] Processing DeleteTablet for tablet 4876c507b5e4457ab4693f674db59207 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:10:00 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:51820
I20260429 14:10:00.767937 22090 tablet_service.cc:1558] Processing DeleteTablet for tablet 4876c507b5e4457ab4693f674db59207 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:10:00 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:41836
I20260429 14:10:00.767961 21954 tablet_service.cc:1558] Processing DeleteTablet for tablet 4876c507b5e4457ab4693f674db59207 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:10:00 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:48422
I20260429 14:10:00.768246 22199 tablet_replica.cc:333] T 4876c507b5e4457ab4693f674db59207 P 81f9542f59404c25b26b1c0556af6fc7: stopping tablet replica
I20260429 14:10:00.768355 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 21755
I20260429 14:10:00.768395 22199 raft_consensus.cc:2243] T 4876c507b5e4457ab4693f674db59207 P 81f9542f59404c25b26b1c0556af6fc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:10:00.768500 22199 raft_consensus.cc:2272] T 4876c507b5e4457ab4693f674db59207 P 81f9542f59404c25b26b1c0556af6fc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:10:00.769079 22199 ts_tablet_manager.cc:1916] T 4876c507b5e4457ab4693f674db59207 P 81f9542f59404c25b26b1c0556af6fc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:10:00.771106 22199 ts_tablet_manager.cc:1929] T 4876c507b5e4457ab4693f674db59207 P 81f9542f59404c25b26b1c0556af6fc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 14:10:00.771174 22199 log.cc:1199] T 4876c507b5e4457ab4693f674db59207 P 81f9542f59404c25b26b1c0556af6fc7: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-1/wal/wals/4876c507b5e4457ab4693f674db59207
I20260429 14:10:00.771649 22199 ts_tablet_manager.cc:1950] T 4876c507b5e4457ab4693f674db59207 P 81f9542f59404c25b26b1c0556af6fc7: Deleting consensus metadata
I20260429 14:10:00.772091 21675 catalog_manager.cc:5002] TS 81f9542f59404c25b26b1c0556af6fc7 (127.2.197.194:38155): tablet 4876c507b5e4457ab4693f674db59207 (table test-table [id=b4d1730a63e34710b4305a986f8648b2]) successfully deleted
W20260429 14:10:00.774633 21675 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.197.193:34723 (error 108)
I20260429 14:10:00.774864 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 21891
I20260429 14:10:00.775926 22169 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:00.775794 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:34723 (local address 127.0.0.1:51840)
0429 14:10:00.775848 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:00.775883 (+ 35us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.193:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260429 14:10:00.776031 21675 catalog_manager.cc:4729] TS 7c1561d59ab741fb933410a67f83a58c (127.2.197.193:34723): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4876c507b5e4457ab4693f674db59207: Network error: Client connection negotiation failed: client connection to 127.2.197.193:34723: connect: Connection refused (error 111)
I20260429 14:10:00.776170 22198 tablet_replica.cc:333] T 4876c507b5e4457ab4693f674db59207 P c5c05b84f90f4495870099021214efcd: stopping tablet replica
I20260429 14:10:00.776275 22198 raft_consensus.cc:2243] T 4876c507b5e4457ab4693f674db59207 P c5c05b84f90f4495870099021214efcd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:10:00.776379 22198 raft_consensus.cc:2272] T 4876c507b5e4457ab4693f674db59207 P c5c05b84f90f4495870099021214efcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:10:00.776894 22198 ts_tablet_manager.cc:1916] T 4876c507b5e4457ab4693f674db59207 P c5c05b84f90f4495870099021214efcd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:10:00.778512 22198 ts_tablet_manager.cc:1929] T 4876c507b5e4457ab4693f674db59207 P c5c05b84f90f4495870099021214efcd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 14:10:00.778604 22198 log.cc:1199] T 4876c507b5e4457ab4693f674db59207 P c5c05b84f90f4495870099021214efcd: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777471413505459-2839-0/minicluster-data/ts-2/wal/wals/4876c507b5e4457ab4693f674db59207
I20260429 14:10:00.778893 22198 ts_tablet_manager.cc:1950] T 4876c507b5e4457ab4693f674db59207 P c5c05b84f90f4495870099021214efcd: Deleting consensus metadata
I20260429 14:10:00.779769 21675 catalog_manager.cc:5002] TS c5c05b84f90f4495870099021214efcd (127.2.197.195:36471): tablet 4876c507b5e4457ab4693f674db59207 (table test-table [id=b4d1730a63e34710b4305a986f8648b2]) successfully deleted
I20260429 14:10:00.780316 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 22027
I20260429 14:10:00.785763 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 21659
2026-04-29T14:10:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (3021 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-04-29T14:10:00Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:10:00Z Disabled control of system clock
I20260429 14:10:00.816015 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:44601
--webserver_interface=127.2.197.254
--webserver_port=0
--builtin_ntp_servers=127.2.197.212:45837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:44601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260429 14:10:00.904887 22204 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:00.905093 22204 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:00.905117 22204 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:10:00.906718 22204 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:10:00.906769 22204 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:00.906781 22204 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:00.906795 22204 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:10:00.906807 22204 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:10:00.908596 22204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:45837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:44601
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:44601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:00.908815 22204 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:00.909027 22204 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:10:00.912339 22210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:00.912403 22212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:00.912601 22204 server_base.cc:1061] running on GCE node
W20260429 14:10:00.912706 22209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:00.912955 22204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:00.913225 22204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:00.914495 22204 hybrid_clock.cc:648] HybridClock initialized: now 1777471800914445 us; error 57 us; skew 500 ppm
I20260429 14:10:00.916193 22204 webserver.cc:492] Webserver started at http://127.2.197.254:41221/ using document root <none> and password file <none>
I20260429 14:10:00.916543 22204 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:00.916630 22204 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:00.916752 22204 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:00.917735 22204 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "65421eba94274f77b7b7792ac5104ba1"
format_stamp: "Formatted at 2026-04-29 14:10:00 on dist-test-slave-8n77"
server_key: "d6df74b1468b2d819da5e9cee8578b2a"
server_key_iv: "6f9932df48d746fbc07fb25deadfd82c"
server_key_version: "encryptionkey@0"
I20260429 14:10:00.918081 22204 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "65421eba94274f77b7b7792ac5104ba1"
format_stamp: "Formatted at 2026-04-29 14:10:00 on dist-test-slave-8n77"
server_key: "d6df74b1468b2d819da5e9cee8578b2a"
server_key_iv: "6f9932df48d746fbc07fb25deadfd82c"
server_key_version: "encryptionkey@0"
I20260429 14:10:00.919471 22204 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260429 14:10:00.920454 22218 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:00.920769 22204 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:10:00.920868 22204 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "65421eba94274f77b7b7792ac5104ba1"
format_stamp: "Formatted at 2026-04-29 14:10:00 on dist-test-slave-8n77"
server_key: "d6df74b1468b2d819da5e9cee8578b2a"
server_key_iv: "6f9932df48d746fbc07fb25deadfd82c"
server_key_version: "encryptionkey@0"
I20260429 14:10:00.920953 22204 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:00.950872 22204 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:00.951253 22204 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:00.951401 22204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:00.955374 22204 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:44601
I20260429 14:10:00.955422 22270 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:44601 every 8 connection(s)
I20260429 14:10:00.955747 22204 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:10:00.956312 22271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:00.958622 22271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65421eba94274f77b7b7792ac5104ba1: Bootstrap starting.
I20260429 14:10:00.959146 22271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65421eba94274f77b7b7792ac5104ba1: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:00.959352 22271 log.cc:826] T 00000000000000000000000000000000 P 65421eba94274f77b7b7792ac5104ba1: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:00.959965 22271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65421eba94274f77b7b7792ac5104ba1: No bootstrap required, opened a new log
I20260429 14:10:00.961108 22271 raft_consensus.cc:359] T 00000000000000000000000000000000 P 65421eba94274f77b7b7792ac5104ba1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65421eba94274f77b7b7792ac5104ba1" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 44601 } }
I20260429 14:10:00.961279 22271 raft_consensus.cc:385] T 00000000000000000000000000000000 P 65421eba94274f77b7b7792ac5104ba1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:10:00.961328 22271 raft_consensus.cc:740] T 00000000000000000000000000000000 P 65421eba94274f77b7b7792ac5104ba1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65421eba94274f77b7b7792ac5104ba1, State: Initialized, Role: FOLLOWER
I20260429 14:10:00.961473 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 22204
I20260429 14:10:00.961467 22271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65421eba94274f77b7b7792ac5104ba1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65421eba94274f77b7b7792ac5104ba1" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 44601 } }
I20260429 14:10:00.961553 22271 raft_consensus.cc:399] T 00000000000000000000000000000000 P 65421eba94274f77b7b7792ac5104ba1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:10:00.961598 22271 raft_consensus.cc:493] T 00000000000000000000000000000000 P 65421eba94274f77b7b7792ac5104ba1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:10:00.961606 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:10:00.961643 22271 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 65421eba94274f77b7b7792ac5104ba1 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:10:00.961781 2839 external_mini_cluster.cc:1468] Setting key fcf55e9b6ca107abb78fc3e4c27da100
I20260429 14:10:00.962469 22271 raft_consensus.cc:515] T 00000000000000000000000000000000 P 65421eba94274f77b7b7792ac5104ba1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65421eba94274f77b7b7792ac5104ba1" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 44601 } }
I20260429 14:10:00.962589 22271 leader_election.cc:304] T 00000000000000000000000000000000 P 65421eba94274f77b7b7792ac5104ba1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65421eba94274f77b7b7792ac5104ba1; no voters:
I20260429 14:10:00.962726 22271 leader_election.cc:290] T 00000000000000000000000000000000 P 65421eba94274f77b7b7792ac5104ba1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:10:00.962805 22276 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 65421eba94274f77b7b7792ac5104ba1 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:10:00.962934 22276 raft_consensus.cc:697] T 00000000000000000000000000000000 P 65421eba94274f77b7b7792ac5104ba1 [term 1 LEADER]: Becoming Leader. State: Replica: 65421eba94274f77b7b7792ac5104ba1, State: Running, Role: LEADER
I20260429 14:10:00.962940 22271 sys_catalog.cc:565] T 00000000000000000000000000000000 P 65421eba94274f77b7b7792ac5104ba1 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:10:00.963039 22276 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65421eba94274f77b7b7792ac5104ba1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65421eba94274f77b7b7792ac5104ba1" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 44601 } }
I20260429 14:10:00.963339 22276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65421eba94274f77b7b7792ac5104ba1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65421eba94274f77b7b7792ac5104ba1. Latest consensus state: current_term: 1 leader_uuid: "65421eba94274f77b7b7792ac5104ba1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65421eba94274f77b7b7792ac5104ba1" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 44601 } } }
I20260429 14:10:00.963344 22277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65421eba94274f77b7b7792ac5104ba1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65421eba94274f77b7b7792ac5104ba1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65421eba94274f77b7b7792ac5104ba1" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 44601 } } }
I20260429 14:10:00.963425 22276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65421eba94274f77b7b7792ac5104ba1 [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:00.963436 22277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65421eba94274f77b7b7792ac5104ba1 [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:00.963708 22280 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:10:00.964263 22280 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:10:00.967159 22275 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:00.962352 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36376 (local address 127.2.197.254:44601)
0429 14:10:00.962579 (+ 227us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:00.962586 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:00.962609 (+ 23us) server_negotiation.cc:408] Connection header received
0429 14:10:00.962713 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:00.962720 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:00.962776 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:00.962907 (+ 131us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:00.963675 (+ 768us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.965186 (+ 1511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:00.965810 (+ 624us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:00.966038 (+ 228us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:00.966447 (+ 409us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:00.966454 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:00.966454 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:10:00.966680 (+ 226us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:00.966702 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:10:00.966710 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:00.966775 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:00.966904 (+ 129us) server_negotiation.cc:300] Negotiation successful
0429 14:10:00.966968 (+ 64us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":121,"threads_started":1}
I20260429 14:10:00.967594 22280 catalog_manager.cc:1357] Generated new cluster ID: f0dfd6c63101408483a7a3bf3392a203
I20260429 14:10:00.967662 22280 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:10:00.982306 22280 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:10:00.982849 22280 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:10:00.988827 22280 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 65421eba94274f77b7b7792ac5104ba1: Generated new TSK 0
I20260429 14:10:00.989022 22280 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 14:10:00.996739 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.193:0
--local_ip_for_outbound_sockets=127.2.197.193
--webserver_interface=127.2.197.193
--webserver_port=0
--tserver_master_addrs=127.2.197.254:44601
--builtin_ntp_servers=127.2.197.212:45837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260429 14:10:01.081844 22295 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:01.082115 22295 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:01.082213 22295 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:10:01.083881 22295 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:01.083959 22295 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:01.084007 22295 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.193
I20260429 14:10:01.085618 22295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:45837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.197.193
--webserver_port=0
--tserver_master_addrs=127.2.197.254:44601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.193
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:01.085877 22295 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:01.086107 22295 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:10:01.088846 22301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:01.089000 22295 server_base.cc:1061] running on GCE node
W20260429 14:10:01.089135 22300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:01.089147 22303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:01.089414 22295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:01.089650 22295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:01.090829 22295 hybrid_clock.cc:648] HybridClock initialized: now 1777471801090814 us; error 31 us; skew 500 ppm
I20260429 14:10:01.091965 22295 webserver.cc:492] Webserver started at http://127.2.197.193:37881/ using document root <none> and password file <none>
I20260429 14:10:01.092176 22295 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:01.092243 22295 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:01.092360 22295 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:01.093353 22295 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-0/data/instance:
uuid: "a0238296c4a14cfc971b48ad960967cf"
format_stamp: "Formatted at 2026-04-29 14:10:01 on dist-test-slave-8n77"
server_key: "b03ac5e0c4cfdae481cfe1d5f84213ec"
server_key_iv: "0101f683dc012d57ca49f3ca6ea9bbed"
server_key_version: "encryptionkey@0"
I20260429 14:10:01.093695 22295 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance:
uuid: "a0238296c4a14cfc971b48ad960967cf"
format_stamp: "Formatted at 2026-04-29 14:10:01 on dist-test-slave-8n77"
server_key: "b03ac5e0c4cfdae481cfe1d5f84213ec"
server_key_iv: "0101f683dc012d57ca49f3ca6ea9bbed"
server_key_version: "encryptionkey@0"
I20260429 14:10:01.095060 22295 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260429 14:10:01.096101 22309 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:01.096374 22295 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:10:01.096464 22295 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-0/wal
uuid: "a0238296c4a14cfc971b48ad960967cf"
format_stamp: "Formatted at 2026-04-29 14:10:01 on dist-test-slave-8n77"
server_key: "b03ac5e0c4cfdae481cfe1d5f84213ec"
server_key_iv: "0101f683dc012d57ca49f3ca6ea9bbed"
server_key_version: "encryptionkey@0"
I20260429 14:10:01.096552 22295 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:01.124228 22295 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:01.124612 22295 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:01.124801 22295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:01.125033 22295 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:10:01.125360 22295 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:10:01.125414 22295 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:01.125460 22295 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:10:01.125485 22295 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:01.131234 22295 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.193:37821
I20260429 14:10:01.131373 22422 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.193:37821 every 8 connection(s)
I20260429 14:10:01.131628 22295 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
I20260429 14:10:01.131990 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 22295
I20260429 14:10:01.132113 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance
I20260429 14:10:01.132326 2839 external_mini_cluster.cc:1468] Setting key 9a10efcaeee5f0ceabe5cbffd26839c6
I20260429 14:10:01.133312 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.194:0
--local_ip_for_outbound_sockets=127.2.197.194
--webserver_interface=127.2.197.194
--webserver_port=0
--tserver_master_addrs=127.2.197.254:44601
--builtin_ntp_servers=127.2.197.212:45837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260429 14:10:01.137552 22275 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:01.132499 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:54379 (local address 127.2.197.254:44601)
0429 14:10:01.132547 (+ 48us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:01.132549 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:01.132817 (+ 268us) server_negotiation.cc:408] Connection header received
0429 14:10:01.133625 (+ 808us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:01.133627 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:01.133645 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:01.133685 (+ 40us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:01.134430 (+ 745us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:01.135270 (+ 840us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:01.136227 (+ 957us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:01.136409 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:01.137130 (+ 721us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:01.137134 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:01.137134 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:10:01.137140 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:01.137156 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:10:01.137159 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:01.137216 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:01.137413 (+ 197us) server_negotiation.cc:300] Negotiation successful
0429 14:10:01.137464 (+ 51us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260429 14:10:01.137733 22425 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:01.132500 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:44601 (local address 127.2.197.193:54379)
0429 14:10:01.132697 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:01.132729 (+ 32us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:01.133421 (+ 692us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:01.133868 (+ 447us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:01.133872 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:01.133884 (+ 12us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:10:01.134329 (+ 445us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:01.134333 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:01.135382 (+ 1049us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:01.135384 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:01.136067 (+ 683us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:01.136070 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:01.136205 (+ 135us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:01.136947 (+ 742us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 14:10:01.136948 (+ 1us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:10:01.136967 (+ 19us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 14:10:01.136995 (+ 28us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:10:01.137246 (+ 251us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:10:01.137252 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:10:01.137256 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 14:10:01.137362 (+ 106us) client_negotiation.cc:241] Negotiation successful
0429 14:10:01.137506 (+ 144us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":88,"threads_started":1}
I20260429 14:10:01.138994 22423 heartbeater.cc:344] Connected to a master server at 127.2.197.254:44601
I20260429 14:10:01.139112 22423 heartbeater.cc:461] Registering TS with master...
I20260429 14:10:01.139324 22423 heartbeater.cc:507] Master 127.2.197.254:44601 requested a full tablet report, sending...
I20260429 14:10:01.139926 22235 ts_manager.cc:194] Registered new tserver with Master: a0238296c4a14cfc971b48ad960967cf (127.2.197.193:37821)
I20260429 14:10:01.141049 22235 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.2.197.193:54379
W20260429 14:10:01.215898 22426 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:01.216113 22426 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:01.216137 22426 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:10:01.218231 22426 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:01.218334 22426 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:01.218367 22426 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.194
I20260429 14:10:01.220130 22426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:45837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.197.194
--webserver_port=0
--tserver_master_addrs=127.2.197.254:44601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.194
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:01.220381 22426 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:01.220580 22426 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:10:01.223418 22434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:01.223423 22432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:01.223651 22426 server_base.cc:1061] running on GCE node
W20260429 14:10:01.223683 22431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:01.224063 22426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:01.224349 22426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:01.225538 22426 hybrid_clock.cc:648] HybridClock initialized: now 1777471801225519 us; error 28 us; skew 500 ppm
I20260429 14:10:01.226825 22426 webserver.cc:492] Webserver started at http://127.2.197.194:44465/ using document root <none> and password file <none>
I20260429 14:10:01.227073 22426 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:01.227164 22426 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:01.227298 22426 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:01.228276 22426 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-1/data/instance:
uuid: "de04f01b46694b3ebe1721ba55bd0e51"
format_stamp: "Formatted at 2026-04-29 14:10:01 on dist-test-slave-8n77"
server_key: "892755269977503c2b0ac341bca4b853"
server_key_iv: "dd94cb1e2b9a12a95d4d8835f92c9606"
server_key_version: "encryptionkey@0"
I20260429 14:10:01.228603 22426 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance:
uuid: "de04f01b46694b3ebe1721ba55bd0e51"
format_stamp: "Formatted at 2026-04-29 14:10:01 on dist-test-slave-8n77"
server_key: "892755269977503c2b0ac341bca4b853"
server_key_iv: "dd94cb1e2b9a12a95d4d8835f92c9606"
server_key_version: "encryptionkey@0"
I20260429 14:10:01.229809 22426 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260429 14:10:01.230686 22440 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:01.230860 22426 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:01.230938 22426 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-1/wal
uuid: "de04f01b46694b3ebe1721ba55bd0e51"
format_stamp: "Formatted at 2026-04-29 14:10:01 on dist-test-slave-8n77"
server_key: "892755269977503c2b0ac341bca4b853"
server_key_iv: "dd94cb1e2b9a12a95d4d8835f92c9606"
server_key_version: "encryptionkey@0"
I20260429 14:10:01.231019 22426 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:01.238540 22426 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:01.238832 22426 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:01.238968 22426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:01.239187 22426 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:10:01.239627 22426 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:10:01.239703 22426 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:01.239775 22426 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:10:01.239806 22426 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:01.245713 22426 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.194:44921
I20260429 14:10:01.245756 22553 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.194:44921 every 8 connection(s)
I20260429 14:10:01.246050 22426 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
I20260429 14:10:01.248682 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 22426
I20260429 14:10:01.248816 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance
I20260429 14:10:01.249020 2839 external_mini_cluster.cc:1468] Setting key a30d7f0cb35d7a160120e96b968e9279
I20260429 14:10:01.249969 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.195:0
--local_ip_for_outbound_sockets=127.2.197.195
--webserver_interface=127.2.197.195
--webserver_port=0
--tserver_master_addrs=127.2.197.254:44601
--builtin_ntp_servers=127.2.197.212:45837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260429 14:10:01.250494 22275 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:01.246809 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:48777 (local address 127.2.197.254:44601)
0429 14:10:01.246876 (+ 67us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:01.246905 (+ 29us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:01.247108 (+ 203us) server_negotiation.cc:408] Connection header received
0429 14:10:01.247654 (+ 546us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:01.247656 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:01.247668 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:01.247736 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:01.248172 (+ 436us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:01.248679 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:01.249426 (+ 747us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:01.249597 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:01.250184 (+ 587us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:01.250187 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:01.250190 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:10:01.250200 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:01.250211 (+ 11us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:10:01.250215 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:01.250305 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:01.250386 (+ 81us) server_negotiation.cc:300] Negotiation successful
0429 14:10:01.250419 (+ 33us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260429 14:10:01.250626 22556 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:01.246789 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:44601 (local address 127.2.197.194:48777)
0429 14:10:01.246988 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:01.247013 (+ 25us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:01.247547 (+ 534us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:01.247797 (+ 250us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:01.247806 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:01.247814 (+ 8us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:10:01.248080 (+ 266us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:01.248082 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:01.248773 (+ 691us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:01.248774 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:01.249328 (+ 554us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:01.249330 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:01.249461 (+ 131us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:01.250021 (+ 560us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 14:10:01.250022 (+ 1us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:10:01.250035 (+ 13us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 14:10:01.250061 (+ 26us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:10:01.250309 (+ 248us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:10:01.250313 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:10:01.250316 (+ 3us) client_negotiation.cc:770] Sending connection context
0429 14:10:01.250392 (+ 76us) client_negotiation.cc:241] Negotiation successful
0429 14:10:01.250457 (+ 65us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":81,"threads_started":1}
I20260429 14:10:01.251061 22554 heartbeater.cc:344] Connected to a master server at 127.2.197.254:44601
I20260429 14:10:01.251158 22554 heartbeater.cc:461] Registering TS with master...
I20260429 14:10:01.251368 22554 heartbeater.cc:507] Master 127.2.197.254:44601 requested a full tablet report, sending...
I20260429 14:10:01.251772 22235 ts_manager.cc:194] Registered new tserver with Master: de04f01b46694b3ebe1721ba55bd0e51 (127.2.197.194:44921)
I20260429 14:10:01.252163 22235 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.2.197.194:48777
W20260429 14:10:01.326800 22557 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:01.327005 22557 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:01.327041 22557 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:10:01.328634 22557 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:01.328706 22557 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:01.328751 22557 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.195
I20260429 14:10:01.330363 22557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:45837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.197.195
--webserver_port=0
--tserver_master_addrs=127.2.197.254:44601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.195
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:01.330621 22557 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:01.330859 22557 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:10:01.333647 22562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:01.333648 22563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:01.333647 22565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:01.333961 22557 server_base.cc:1061] running on GCE node
I20260429 14:10:01.334167 22557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:01.334468 22557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:01.335649 22557 hybrid_clock.cc:648] HybridClock initialized: now 1777471801335629 us; error 31 us; skew 500 ppm
I20260429 14:10:01.336907 22557 webserver.cc:492] Webserver started at http://127.2.197.195:43997/ using document root <none> and password file <none>
I20260429 14:10:01.337167 22557 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:01.337312 22557 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:01.337456 22557 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:01.338546 22557 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-2/data/instance:
uuid: "bf4709df030f46f8a917017416b009e3"
format_stamp: "Formatted at 2026-04-29 14:10:01 on dist-test-slave-8n77"
server_key: "44dbbce26e2a8638aecf5cb1dac97464"
server_key_iv: "2b10b480b0fc158320d88bb4f49249bf"
server_key_version: "encryptionkey@0"
I20260429 14:10:01.338896 22557 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance:
uuid: "bf4709df030f46f8a917017416b009e3"
format_stamp: "Formatted at 2026-04-29 14:10:01 on dist-test-slave-8n77"
server_key: "44dbbce26e2a8638aecf5cb1dac97464"
server_key_iv: "2b10b480b0fc158320d88bb4f49249bf"
server_key_version: "encryptionkey@0"
I20260429 14:10:01.340238 22557 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260429 14:10:01.341159 22571 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:01.341537 22557 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:10:01.341653 22557 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-2/wal
uuid: "bf4709df030f46f8a917017416b009e3"
format_stamp: "Formatted at 2026-04-29 14:10:01 on dist-test-slave-8n77"
server_key: "44dbbce26e2a8638aecf5cb1dac97464"
server_key_iv: "2b10b480b0fc158320d88bb4f49249bf"
server_key_version: "encryptionkey@0"
I20260429 14:10:01.341740 22557 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:01.363725 22557 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:01.364069 22557 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:01.364217 22557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:01.364459 22557 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:10:01.364832 22557 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:10:01.364938 22557 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:01.364984 22557 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:10:01.365007 22557 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:01.371232 22557 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.195:40255
I20260429 14:10:01.371438 22684 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.195:40255 every 8 connection(s)
I20260429 14:10:01.371636 22557 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
I20260429 14:10:01.374933 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 22557
I20260429 14:10:01.375068 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance
I20260429 14:10:01.375283 2839 external_mini_cluster.cc:1468] Setting key 6ef196c84400ac1284e5769bf0e35e4e
I20260429 14:10:01.376214 22275 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:01.372514 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:46815 (local address 127.2.197.254:44601)
0429 14:10:01.372605 (+ 91us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:01.372608 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:01.372754 (+ 146us) server_negotiation.cc:408] Connection header received
0429 14:10:01.373274 (+ 520us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:01.373275 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:01.373287 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:01.373335 (+ 48us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:01.373789 (+ 454us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:01.374450 (+ 661us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:01.375013 (+ 563us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:01.375185 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:01.375878 (+ 693us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:01.375882 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:01.375882 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:10:01.375888 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:01.375901 (+ 13us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:10:01.375924 (+ 23us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:01.375995 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:01.376095 (+ 100us) server_negotiation.cc:300] Negotiation successful
0429 14:10:01.376131 (+ 36us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260429 14:10:01.376699 22687 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:01.372398 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:44601 (local address 127.2.197.195:46815)
0429 14:10:01.372649 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:01.372676 (+ 27us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:01.373170 (+ 494us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:01.373418 (+ 248us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:01.373422 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:01.373429 (+ 7us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:10:01.373697 (+ 268us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:01.373699 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:01.374540 (+ 841us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:01.374541 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:01.374926 (+ 385us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:01.374928 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:01.375063 (+ 135us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:01.375713 (+ 650us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 14:10:01.375714 (+ 1us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:10:01.375729 (+ 15us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 14:10:01.375752 (+ 23us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:10:01.376002 (+ 250us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:10:01.376006 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:10:01.376009 (+ 3us) client_negotiation.cc:770] Sending connection context
0429 14:10:01.376087 (+ 78us) client_negotiation.cc:241] Negotiation successful
0429 14:10:01.376156 (+ 69us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":100,"threads_started":1}
I20260429 14:10:01.377310 22685 heartbeater.cc:344] Connected to a master server at 127.2.197.254:44601
I20260429 14:10:01.377410 22685 heartbeater.cc:461] Registering TS with master...
I20260429 14:10:01.377564 22685 heartbeater.cc:507] Master 127.2.197.254:44601 requested a full tablet report, sending...
I20260429 14:10:01.377948 22235 ts_manager.cc:194] Registered new tserver with Master: bf4709df030f46f8a917017416b009e3 (127.2.197.195:40255)
I20260429 14:10:01.378345 22235 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.2.197.195:46815
I20260429 14:10:01.387416 2839 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260429 14:10:01.393976 22275 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:01.391011 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36382 (local address 127.2.197.254:44601)
0429 14:10:01.391081 (+ 70us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:01.391083 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:01.391327 (+ 244us) server_negotiation.cc:408] Connection header received
0429 14:10:01.391359 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:01.391360 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:01.391371 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:01.391432 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:01.391789 (+ 357us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:01.392296 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:01.392900 (+ 604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:01.393024 (+ 124us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:01.393578 (+ 554us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:01.393581 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:01.393581 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:10:01.393586 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:01.393597 (+ 11us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:10:01.393600 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:01.393665 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:01.393859 (+ 194us) server_negotiation.cc:300] Negotiation successful
0429 14:10:01.393878 (+ 19us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260429 14:10:01.395892 22235 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36382:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 14:10:01.397518 22235 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 14:10:01.404036 22696 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:01.399715 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:44921 (local address 127.0.0.1:51730)
0429 14:10:01.400083 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:01.400192 (+ 109us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:01.400281 (+ 89us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:01.400671 (+ 390us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:01.400674 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:01.400677 (+ 3us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:10:01.400945 (+ 268us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:01.400947 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:01.402782 (+ 1835us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:01.402783 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:01.403748 (+ 965us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:01.403751 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:01.403831 (+ 80us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:01.403868 (+ 37us) client_negotiation.cc:770] Sending connection context
0429 14:10:01.403920 (+ 52us) client_negotiation.cc:241] Negotiation successful
0429 14:10:01.403957 (+ 37us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":86,"threads_started":1}
I20260429 14:10:01.404644 22697 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:01.399974 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51730 (local address 127.2.197.194:44921)
0429 14:10:01.400406 (+ 432us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:01.400416 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:01.400481 (+ 65us) server_negotiation.cc:408] Connection header received
0429 14:10:01.400528 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:01.400530 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:01.400551 (+ 21us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:01.400635 (+ 84us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:10:01.401014 (+ 379us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:01.402697 (+ 1683us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:01.403847 (+ 1150us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:01.404438 (+ 591us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:01.404526 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:01.404555 (+ 29us) server_negotiation.cc:300] Negotiation successful
0429 14:10:01.404586 (+ 31us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":83,"threads_started":1}
I20260429 14:10:01.405211 22488 tablet_service.cc:1511] Processing CreateTablet for tablet 89ed08d9c43a4cbf93aa1839c67dc16f (DEFAULT_TABLE table=test-table [id=dce8148ae323454bae6c6e9616b4d399]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:10:01.405378 22693 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:01.399716 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:40255 (local address 127.0.0.1:33248)
0429 14:10:01.400304 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:01.400319 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:01.400653 (+ 334us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:01.400901 (+ 248us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:01.400902 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:01.400904 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:10:01.401087 (+ 183us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:01.401089 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:01.404239 (+ 3150us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:01.404239 (+ 0us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:01.405163 (+ 924us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:01.405165 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:01.405234 (+ 69us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:01.405236 (+ 2us) client_negotiation.cc:770] Sending connection context
0429 14:10:01.405271 (+ 35us) client_negotiation.cc:241] Negotiation successful
0429 14:10:01.405307 (+ 36us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":2,"thread_start_us":86,"threads_started":1}
I20260429 14:10:01.405526 22488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89ed08d9c43a4cbf93aa1839c67dc16f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:01.406075 22698 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:01.400032 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54218 (local address 127.2.197.193:37821)
0429 14:10:01.400416 (+ 384us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:01.400426 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:01.400581 (+ 155us) server_negotiation.cc:408] Connection header received
0429 14:10:01.401055 (+ 474us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:01.401058 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:01.401082 (+ 24us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:01.401148 (+ 66us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:10:01.402647 (+ 1499us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:01.404124 (+ 1477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:01.405347 (+ 1223us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:01.405854 (+ 507us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:01.405932 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:01.405965 (+ 33us) server_negotiation.cc:300] Negotiation successful
0429 14:10:01.406002 (+ 37us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":78,"threads_started":1}
I20260429 14:10:01.406237 22694 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:01.399951 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:37821 (local address 127.0.0.1:54218)
0429 14:10:01.400483 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:01.400495 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:01.400576 (+ 81us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:01.402388 (+ 1812us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:01.402388 (+ 0us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:01.402391 (+ 3us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:10:01.402532 (+ 141us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:01.402534 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:01.404219 (+ 1685us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:01.404220 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:01.405171 (+ 951us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:01.405172 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:01.405232 (+ 60us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:01.405236 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 14:10:01.405271 (+ 35us) client_negotiation.cc:241] Negotiation successful
0429 14:10:01.405313 (+ 42us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":69,"threads_started":1}
I20260429 14:10:01.406246 22695 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:01.399850 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33248 (local address 127.2.197.195:40255)
0429 14:10:01.400067 (+ 217us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:01.400075 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:01.400424 (+ 349us) server_negotiation.cc:408] Connection header received
0429 14:10:01.400771 (+ 347us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:01.400773 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:01.400818 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:01.400888 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:10:01.401167 (+ 279us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:01.402781 (+ 1614us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:01.405537 (+ 2756us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:01.406028 (+ 491us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:01.406090 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:01.406123 (+ 33us) server_negotiation.cc:300] Negotiation successful
0429 14:10:01.406174 (+ 51us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":117,"threads_started":1}
I20260429 14:10:01.406952 22619 tablet_service.cc:1511] Processing CreateTablet for tablet 89ed08d9c43a4cbf93aa1839c67dc16f (DEFAULT_TABLE table=test-table [id=dce8148ae323454bae6c6e9616b4d399]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:10:01.407184 22619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89ed08d9c43a4cbf93aa1839c67dc16f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:01.407202 22357 tablet_service.cc:1511] Processing CreateTablet for tablet 89ed08d9c43a4cbf93aa1839c67dc16f (DEFAULT_TABLE table=test-table [id=dce8148ae323454bae6c6e9616b4d399]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:10:01.407478 22357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89ed08d9c43a4cbf93aa1839c67dc16f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:01.408784 22699 tablet_bootstrap.cc:492] T 89ed08d9c43a4cbf93aa1839c67dc16f P de04f01b46694b3ebe1721ba55bd0e51: Bootstrap starting.
I20260429 14:10:01.409407 22699 tablet_bootstrap.cc:654] T 89ed08d9c43a4cbf93aa1839c67dc16f P de04f01b46694b3ebe1721ba55bd0e51: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:01.409718 22699 log.cc:826] T 89ed08d9c43a4cbf93aa1839c67dc16f P de04f01b46694b3ebe1721ba55bd0e51: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:01.410013 22700 tablet_bootstrap.cc:492] T 89ed08d9c43a4cbf93aa1839c67dc16f P bf4709df030f46f8a917017416b009e3: Bootstrap starting.
I20260429 14:10:01.410511 22699 tablet_bootstrap.cc:492] T 89ed08d9c43a4cbf93aa1839c67dc16f P de04f01b46694b3ebe1721ba55bd0e51: No bootstrap required, opened a new log
I20260429 14:10:01.410586 22699 ts_tablet_manager.cc:1403] T 89ed08d9c43a4cbf93aa1839c67dc16f P de04f01b46694b3ebe1721ba55bd0e51: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260429 14:10:01.410749 22702 tablet_bootstrap.cc:492] T 89ed08d9c43a4cbf93aa1839c67dc16f P a0238296c4a14cfc971b48ad960967cf: Bootstrap starting.
I20260429 14:10:01.410908 22700 tablet_bootstrap.cc:654] T 89ed08d9c43a4cbf93aa1839c67dc16f P bf4709df030f46f8a917017416b009e3: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:01.411303 22700 log.cc:826] T 89ed08d9c43a4cbf93aa1839c67dc16f P bf4709df030f46f8a917017416b009e3: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:01.411418 22702 tablet_bootstrap.cc:654] T 89ed08d9c43a4cbf93aa1839c67dc16f P a0238296c4a14cfc971b48ad960967cf: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:01.411731 22702 log.cc:826] T 89ed08d9c43a4cbf93aa1839c67dc16f P a0238296c4a14cfc971b48ad960967cf: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:01.412094 22700 tablet_bootstrap.cc:492] T 89ed08d9c43a4cbf93aa1839c67dc16f P bf4709df030f46f8a917017416b009e3: No bootstrap required, opened a new log
I20260429 14:10:01.412159 22700 ts_tablet_manager.cc:1403] T 89ed08d9c43a4cbf93aa1839c67dc16f P bf4709df030f46f8a917017416b009e3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260429 14:10:01.412264 22699 raft_consensus.cc:359] T 89ed08d9c43a4cbf93aa1839c67dc16f P de04f01b46694b3ebe1721ba55bd0e51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de04f01b46694b3ebe1721ba55bd0e51" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 44921 } } peers { permanent_uuid: "a0238296c4a14cfc971b48ad960967cf" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 37821 } } peers { permanent_uuid: "bf4709df030f46f8a917017416b009e3" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40255 } }
I20260429 14:10:01.412405 22699 raft_consensus.cc:385] T 89ed08d9c43a4cbf93aa1839c67dc16f P de04f01b46694b3ebe1721ba55bd0e51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:10:01.412447 22699 raft_consensus.cc:740] T 89ed08d9c43a4cbf93aa1839c67dc16f P de04f01b46694b3ebe1721ba55bd0e51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de04f01b46694b3ebe1721ba55bd0e51, State: Initialized, Role: FOLLOWER
I20260429 14:10:01.412547 22699 consensus_queue.cc:260] T 89ed08d9c43a4cbf93aa1839c67dc16f P de04f01b46694b3ebe1721ba55bd0e51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de04f01b46694b3ebe1721ba55bd0e51" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 44921 } } peers { permanent_uuid: "a0238296c4a14cfc971b48ad960967cf" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 37821 } } peers { permanent_uuid: "bf4709df030f46f8a917017416b009e3" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40255 } }
I20260429 14:10:01.412866 22554 heartbeater.cc:499] Master 127.2.197.254:44601 was elected leader, sending a full tablet report...
I20260429 14:10:01.412811 22699 ts_tablet_manager.cc:1434] T 89ed08d9c43a4cbf93aa1839c67dc16f P de04f01b46694b3ebe1721ba55bd0e51: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260429 14:10:01.413720 22702 tablet_bootstrap.cc:492] T 89ed08d9c43a4cbf93aa1839c67dc16f P a0238296c4a14cfc971b48ad960967cf: No bootstrap required, opened a new log
I20260429 14:10:01.413789 22702 ts_tablet_manager.cc:1403] T 89ed08d9c43a4cbf93aa1839c67dc16f P a0238296c4a14cfc971b48ad960967cf: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260429 14:10:01.414009 22700 raft_consensus.cc:359] T 89ed08d9c43a4cbf93aa1839c67dc16f P bf4709df030f46f8a917017416b009e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de04f01b46694b3ebe1721ba55bd0e51" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 44921 } } peers { permanent_uuid: "a0238296c4a14cfc971b48ad960967cf" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 37821 } } peers { permanent_uuid: "bf4709df030f46f8a917017416b009e3" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40255 } }
I20260429 14:10:01.414144 22700 raft_consensus.cc:385] T 89ed08d9c43a4cbf93aa1839c67dc16f P bf4709df030f46f8a917017416b009e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:10:01.414183 22700 raft_consensus.cc:740] T 89ed08d9c43a4cbf93aa1839c67dc16f P bf4709df030f46f8a917017416b009e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf4709df030f46f8a917017416b009e3, State: Initialized, Role: FOLLOWER
I20260429 14:10:01.414327 22700 consensus_queue.cc:260] T 89ed08d9c43a4cbf93aa1839c67dc16f P bf4709df030f46f8a917017416b009e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de04f01b46694b3ebe1721ba55bd0e51" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 44921 } } peers { permanent_uuid: "a0238296c4a14cfc971b48ad960967cf" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 37821 } } peers { permanent_uuid: "bf4709df030f46f8a917017416b009e3" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40255 } }
I20260429 14:10:01.414574 22700 ts_tablet_manager.cc:1434] T 89ed08d9c43a4cbf93aa1839c67dc16f P bf4709df030f46f8a917017416b009e3: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260429 14:10:01.414640 22685 heartbeater.cc:499] Master 127.2.197.254:44601 was elected leader, sending a full tablet report...
I20260429 14:10:01.415549 22702 raft_consensus.cc:359] T 89ed08d9c43a4cbf93aa1839c67dc16f P a0238296c4a14cfc971b48ad960967cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de04f01b46694b3ebe1721ba55bd0e51" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 44921 } } peers { permanent_uuid: "a0238296c4a14cfc971b48ad960967cf" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 37821 } } peers { permanent_uuid: "bf4709df030f46f8a917017416b009e3" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40255 } }
I20260429 14:10:01.415653 22702 raft_consensus.cc:385] T 89ed08d9c43a4cbf93aa1839c67dc16f P a0238296c4a14cfc971b48ad960967cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:10:01.415704 22702 raft_consensus.cc:740] T 89ed08d9c43a4cbf93aa1839c67dc16f P a0238296c4a14cfc971b48ad960967cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0238296c4a14cfc971b48ad960967cf, State: Initialized, Role: FOLLOWER
I20260429 14:10:01.415808 22702 consensus_queue.cc:260] T 89ed08d9c43a4cbf93aa1839c67dc16f P a0238296c4a14cfc971b48ad960967cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de04f01b46694b3ebe1721ba55bd0e51" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 44921 } } peers { permanent_uuid: "a0238296c4a14cfc971b48ad960967cf" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 37821 } } peers { permanent_uuid: "bf4709df030f46f8a917017416b009e3" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40255 } }
I20260429 14:10:01.416046 22702 ts_tablet_manager.cc:1434] T 89ed08d9c43a4cbf93aa1839c67dc16f P a0238296c4a14cfc971b48ad960967cf: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260429 14:10:01.416167 22423 heartbeater.cc:499] Master 127.2.197.254:44601 was elected leader, sending a full tablet report...
W20260429 14:10:01.496634 22555 tablet.cc:2404] T 89ed08d9c43a4cbf93aa1839c67dc16f P de04f01b46694b3ebe1721ba55bd0e51: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260429 14:10:01.622455 22686 tablet.cc:2404] T 89ed08d9c43a4cbf93aa1839c67dc16f P bf4709df030f46f8a917017416b009e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260429 14:10:01.632417 22424 tablet.cc:2404] T 89ed08d9c43a4cbf93aa1839c67dc16f P a0238296c4a14cfc971b48ad960967cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 14:10:01.658524 22707 raft_consensus.cc:493] T 89ed08d9c43a4cbf93aa1839c67dc16f P a0238296c4a14cfc971b48ad960967cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:10:01.658704 22707 raft_consensus.cc:515] T 89ed08d9c43a4cbf93aa1839c67dc16f P a0238296c4a14cfc971b48ad960967cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de04f01b46694b3ebe1721ba55bd0e51" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 44921 } } peers { permanent_uuid: "a0238296c4a14cfc971b48ad960967cf" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 37821 } } peers { permanent_uuid: "bf4709df030f46f8a917017416b009e3" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40255 } }
I20260429 14:10:01.659104 22707 leader_election.cc:290] T 89ed08d9c43a4cbf93aa1839c67dc16f P a0238296c4a14cfc971b48ad960967cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de04f01b46694b3ebe1721ba55bd0e51 (127.2.197.194:44921), bf4709df030f46f8a917017416b009e3 (127.2.197.195:40255)
I20260429 14:10:01.661909 22709 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:01.659344 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:40255 (local address 127.2.197.193:51349)
0429 14:10:01.659487 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:01.659500 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:01.659568 (+ 68us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:01.659723 (+ 155us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:01.659724 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:01.659727 (+ 3us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:10:01.659931 (+ 204us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:01.659933 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:01.661018 (+ 1085us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:01.661019 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:01.661641 (+ 622us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:01.661643 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:01.661741 (+ 98us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:01.661745 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 14:10:01.661779 (+ 34us) client_negotiation.cc:241] Negotiation successful
0429 14:10:01.661819 (+ 40us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":76,"threads_started":1}
I20260429 14:10:01.662400 22695 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:01.659440 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:51349 (local address 127.2.197.195:40255)
0429 14:10:01.659497 (+ 57us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:01.659499 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:01.659550 (+ 51us) server_negotiation.cc:408] Connection header received
0429 14:10:01.659629 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:01.659630 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:01.659642 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:01.659718 (+ 76us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:10:01.660035 (+ 317us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:01.660912 (+ 877us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:01.661739 (+ 827us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:01.662244 (+ 505us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:01.662255 (+ 11us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:01.662279 (+ 24us) server_negotiation.cc:300] Negotiation successful
0429 14:10:01.662336 (+ 57us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260429 14:10:01.662719 22639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89ed08d9c43a4cbf93aa1839c67dc16f" candidate_uuid: "a0238296c4a14cfc971b48ad960967cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf4709df030f46f8a917017416b009e3" is_pre_election: true
I20260429 14:10:01.662884 22639 raft_consensus.cc:2468] T 89ed08d9c43a4cbf93aa1839c67dc16f P bf4709df030f46f8a917017416b009e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0238296c4a14cfc971b48ad960967cf in term 0.
I20260429 14:10:01.663179 22311 leader_election.cc:304] T 89ed08d9c43a4cbf93aa1839c67dc16f P a0238296c4a14cfc971b48ad960967cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0238296c4a14cfc971b48ad960967cf, bf4709df030f46f8a917017416b009e3; no voters:
I20260429 14:10:01.663305 22708 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:01.659362 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:44921 (local address 127.2.197.193:57251)
0429 14:10:01.659514 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:01.659526 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:01.659591 (+ 65us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:01.659761 (+ 170us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:01.659762 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:01.659764 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:10:01.660704 (+ 940us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:01.660712 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:01.662217 (+ 1505us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:01.662218 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:01.663069 (+ 851us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:01.663072 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:01.663144 (+ 72us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:01.663146 (+ 2us) client_negotiation.cc:770] Sending connection context
0429 14:10:01.663170 (+ 24us) client_negotiation.cc:241] Negotiation successful
0429 14:10:01.663204 (+ 34us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":82,"threads_started":1}
I20260429 14:10:01.663395 22707 raft_consensus.cc:2804] T 89ed08d9c43a4cbf93aa1839c67dc16f P a0238296c4a14cfc971b48ad960967cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 14:10:01.663455 22707 raft_consensus.cc:493] T 89ed08d9c43a4cbf93aa1839c67dc16f P a0238296c4a14cfc971b48ad960967cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 14:10:01.663477 22707 raft_consensus.cc:3060] T 89ed08d9c43a4cbf93aa1839c67dc16f P a0238296c4a14cfc971b48ad960967cf [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:10:01.663888 22697 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:01.659496 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:57251 (local address 127.2.197.194:44921)
0429 14:10:01.659563 (+ 67us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:01.659566 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:01.659574 (+ 8us) server_negotiation.cc:408] Connection header received
0429 14:10:01.659668 (+ 94us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:01.659669 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:01.659680 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:01.659757 (+ 77us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:10:01.660956 (+ 1199us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:01.662103 (+ 1147us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:01.663164 (+ 1061us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:01.663753 (+ 589us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:01.663762 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:01.663788 (+ 26us) server_negotiation.cc:300] Negotiation successful
0429 14:10:01.663825 (+ 37us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260429 14:10:01.664413 22508 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89ed08d9c43a4cbf93aa1839c67dc16f" candidate_uuid: "a0238296c4a14cfc971b48ad960967cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de04f01b46694b3ebe1721ba55bd0e51" is_pre_election: true
I20260429 14:10:01.664711 22508 raft_consensus.cc:2468] T 89ed08d9c43a4cbf93aa1839c67dc16f P de04f01b46694b3ebe1721ba55bd0e51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0238296c4a14cfc971b48ad960967cf in term 0.
I20260429 14:10:01.665084 22707 raft_consensus.cc:515] T 89ed08d9c43a4cbf93aa1839c67dc16f P a0238296c4a14cfc971b48ad960967cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de04f01b46694b3ebe1721ba55bd0e51" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 44921 } } peers { permanent_uuid: "a0238296c4a14cfc971b48ad960967cf" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 37821 } } peers { permanent_uuid: "bf4709df030f46f8a917017416b009e3" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40255 } }
I20260429 14:10:01.665248 22707 leader_election.cc:290] T 89ed08d9c43a4cbf93aa1839c67dc16f P a0238296c4a14cfc971b48ad960967cf [CANDIDATE]: Term 1 election: Requested vote from peers de04f01b46694b3ebe1721ba55bd0e51 (127.2.197.194:44921), bf4709df030f46f8a917017416b009e3 (127.2.197.195:40255)
I20260429 14:10:01.665408 22508 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89ed08d9c43a4cbf93aa1839c67dc16f" candidate_uuid: "a0238296c4a14cfc971b48ad960967cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de04f01b46694b3ebe1721ba55bd0e51"
I20260429 14:10:01.665498 22508 raft_consensus.cc:3060] T 89ed08d9c43a4cbf93aa1839c67dc16f P de04f01b46694b3ebe1721ba55bd0e51 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:10:01.665498 22639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89ed08d9c43a4cbf93aa1839c67dc16f" candidate_uuid: "a0238296c4a14cfc971b48ad960967cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf4709df030f46f8a917017416b009e3"
I20260429 14:10:01.665630 22639 raft_consensus.cc:3060] T 89ed08d9c43a4cbf93aa1839c67dc16f P bf4709df030f46f8a917017416b009e3 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:10:01.666309 22508 raft_consensus.cc:2468] T 89ed08d9c43a4cbf93aa1839c67dc16f P de04f01b46694b3ebe1721ba55bd0e51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0238296c4a14cfc971b48ad960967cf in term 1.
I20260429 14:10:01.666407 22639 raft_consensus.cc:2468] T 89ed08d9c43a4cbf93aa1839c67dc16f P bf4709df030f46f8a917017416b009e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0238296c4a14cfc971b48ad960967cf in term 1.
I20260429 14:10:01.666551 22313 leader_election.cc:304] T 89ed08d9c43a4cbf93aa1839c67dc16f P a0238296c4a14cfc971b48ad960967cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0238296c4a14cfc971b48ad960967cf, de04f01b46694b3ebe1721ba55bd0e51; no voters:
I20260429 14:10:01.666671 22707 raft_consensus.cc:2804] T 89ed08d9c43a4cbf93aa1839c67dc16f P a0238296c4a14cfc971b48ad960967cf [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:10:01.666824 22707 raft_consensus.cc:697] T 89ed08d9c43a4cbf93aa1839c67dc16f P a0238296c4a14cfc971b48ad960967cf [term 1 LEADER]: Becoming Leader. State: Replica: a0238296c4a14cfc971b48ad960967cf, State: Running, Role: LEADER
I20260429 14:10:01.666931 22707 consensus_queue.cc:237] T 89ed08d9c43a4cbf93aa1839c67dc16f P a0238296c4a14cfc971b48ad960967cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de04f01b46694b3ebe1721ba55bd0e51" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 44921 } } peers { permanent_uuid: "a0238296c4a14cfc971b48ad960967cf" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 37821 } } peers { permanent_uuid: "bf4709df030f46f8a917017416b009e3" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40255 } }
I20260429 14:10:01.667971 22235 catalog_manager.cc:5671] T 89ed08d9c43a4cbf93aa1839c67dc16f P a0238296c4a14cfc971b48ad960967cf reported cstate change: term changed from 0 to 1, leader changed from <none> to a0238296c4a14cfc971b48ad960967cf (127.2.197.193). New cstate: current_term: 1 leader_uuid: "a0238296c4a14cfc971b48ad960967cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de04f01b46694b3ebe1721ba55bd0e51" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 44921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0238296c4a14cfc971b48ad960967cf" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 37821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf4709df030f46f8a917017416b009e3" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40255 } health_report { overall_health: UNKNOWN } } }
I20260429 14:10:01.692683 22698 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:01.689504 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54228 (local address 127.2.197.193:37821)
0429 14:10:01.689579 (+ 75us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:01.689585 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:01.689632 (+ 47us) server_negotiation.cc:408] Connection header received
0429 14:10:01.689854 (+ 222us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:01.689856 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:01.689882 (+ 26us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:01.689948 (+ 66us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:10:01.690343 (+ 395us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:01.690874 (+ 531us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:01.691596 (+ 722us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:01.691792 (+ 196us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:01.692037 (+ 245us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:10:01.692198 (+ 161us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:10:01.692281 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:01.692537 (+ 256us) server_negotiation.cc:300] Negotiation successful
0429 14:10:01.692556 (+ 19us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260429 14:10:01.694752 22508 raft_consensus.cc:1275] T 89ed08d9c43a4cbf93aa1839c67dc16f P de04f01b46694b3ebe1721ba55bd0e51 [term 1 FOLLOWER]: Refusing update from remote peer a0238296c4a14cfc971b48ad960967cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:10:01.694767 22639 raft_consensus.cc:1275] T 89ed08d9c43a4cbf93aa1839c67dc16f P bf4709df030f46f8a917017416b009e3 [term 1 FOLLOWER]: Refusing update from remote peer a0238296c4a14cfc971b48ad960967cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:10:01.695122 22707 consensus_queue.cc:1048] T 89ed08d9c43a4cbf93aa1839c67dc16f P a0238296c4a14cfc971b48ad960967cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf4709df030f46f8a917017416b009e3" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 40255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:10:01.695525 22710 consensus_queue.cc:1048] T 89ed08d9c43a4cbf93aa1839c67dc16f P a0238296c4a14cfc971b48ad960967cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "de04f01b46694b3ebe1721ba55bd0e51" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 44921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:10:01.698787 22713 mvcc.cc:204] Tried to move back new op lower bound from 7280524499738451968 to 7280524499628544000. Current Snapshot: MvccSnapshot[applied={T|T < 7280524499738451968}]
I20260429 14:10:01.698907 22714 mvcc.cc:204] Tried to move back new op lower bound from 7280524499738451968 to 7280524499628544000. Current Snapshot: MvccSnapshot[applied={T|T < 7280524499738451968}]
I20260429 14:10:01.699754 22715 mvcc.cc:204] Tried to move back new op lower bound from 7280524499738451968 to 7280524499628544000. Current Snapshot: MvccSnapshot[applied={T|T < 7280524499738451968}]
I20260429 14:10:01.701180 22233 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:10:01.701359 22233 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:10:01.702587 22233 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 65421eba94274f77b7b7792ac5104ba1: Sending DeleteTablet for 3 replicas of tablet 89ed08d9c43a4cbf93aa1839c67dc16f
I20260429 14:10:01.702966 22619 tablet_service.cc:1558] Processing DeleteTablet for tablet 89ed08d9c43a4cbf93aa1839c67dc16f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:10:01 UTC) from {username='slave'} at 127.0.0.1:33248
I20260429 14:10:01.702948 22488 tablet_service.cc:1558] Processing DeleteTablet for tablet 89ed08d9c43a4cbf93aa1839c67dc16f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:10:01 UTC) from {username='slave'} at 127.0.0.1:51730
I20260429 14:10:01.703210 22357 tablet_service.cc:1558] Processing DeleteTablet for tablet 89ed08d9c43a4cbf93aa1839c67dc16f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:10:01 UTC) from {username='slave'} at 127.0.0.1:54218
I20260429 14:10:01.703248 22723 tablet_replica.cc:333] T 89ed08d9c43a4cbf93aa1839c67dc16f P de04f01b46694b3ebe1721ba55bd0e51: stopping tablet replica
I20260429 14:10:01.703339 22723 raft_consensus.cc:2243] T 89ed08d9c43a4cbf93aa1839c67dc16f P de04f01b46694b3ebe1721ba55bd0e51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:10:01.703444 22723 raft_consensus.cc:2272] T 89ed08d9c43a4cbf93aa1839c67dc16f P de04f01b46694b3ebe1721ba55bd0e51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:10:01.703442 22725 tablet_replica.cc:333] T 89ed08d9c43a4cbf93aa1839c67dc16f P a0238296c4a14cfc971b48ad960967cf: stopping tablet replica
I20260429 14:10:01.703516 22725 raft_consensus.cc:2243] T 89ed08d9c43a4cbf93aa1839c67dc16f P a0238296c4a14cfc971b48ad960967cf [term 1 LEADER]: Raft consensus shutting down.
I20260429 14:10:01.703617 22725 raft_consensus.cc:2272] T 89ed08d9c43a4cbf93aa1839c67dc16f P a0238296c4a14cfc971b48ad960967cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:10:01.703713 22724 tablet_replica.cc:333] T 89ed08d9c43a4cbf93aa1839c67dc16f P bf4709df030f46f8a917017416b009e3: stopping tablet replica
I20260429 14:10:01.703717 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 22295
I20260429 14:10:01.703797 22724 raft_consensus.cc:2243] T 89ed08d9c43a4cbf93aa1839c67dc16f P bf4709df030f46f8a917017416b009e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:10:01.703888 22724 raft_consensus.cc:2272] T 89ed08d9c43a4cbf93aa1839c67dc16f P bf4709df030f46f8a917017416b009e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:10:01.703948 22723 ts_tablet_manager.cc:1916] T 89ed08d9c43a4cbf93aa1839c67dc16f P de04f01b46694b3ebe1721ba55bd0e51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:10:01.704658 22724 ts_tablet_manager.cc:1916] T 89ed08d9c43a4cbf93aa1839c67dc16f P bf4709df030f46f8a917017416b009e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:10:01.706004 22723 ts_tablet_manager.cc:1929] T 89ed08d9c43a4cbf93aa1839c67dc16f P de04f01b46694b3ebe1721ba55bd0e51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 14:10:01.706075 22723 log.cc:1199] T 89ed08d9c43a4cbf93aa1839c67dc16f P de04f01b46694b3ebe1721ba55bd0e51: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-1/wal/wals/89ed08d9c43a4cbf93aa1839c67dc16f
I20260429 14:10:01.706422 22723 ts_tablet_manager.cc:1950] T 89ed08d9c43a4cbf93aa1839c67dc16f P de04f01b46694b3ebe1721ba55bd0e51: Deleting consensus metadata
I20260429 14:10:01.706523 22724 ts_tablet_manager.cc:1929] T 89ed08d9c43a4cbf93aa1839c67dc16f P bf4709df030f46f8a917017416b009e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 14:10:01.706591 22724 log.cc:1199] T 89ed08d9c43a4cbf93aa1839c67dc16f P bf4709df030f46f8a917017416b009e3: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777471413505459-2839-0/minicluster-data/ts-2/wal/wals/89ed08d9c43a4cbf93aa1839c67dc16f
I20260429 14:10:01.706880 22724 ts_tablet_manager.cc:1950] T 89ed08d9c43a4cbf93aa1839c67dc16f P bf4709df030f46f8a917017416b009e3: Deleting consensus metadata
I20260429 14:10:01.707173 22222 catalog_manager.cc:5002] TS de04f01b46694b3ebe1721ba55bd0e51 (127.2.197.194:44921): tablet 89ed08d9c43a4cbf93aa1839c67dc16f (table test-table [id=dce8148ae323454bae6c6e9616b4d399]) successfully deleted
I20260429 14:10:01.707757 22220 catalog_manager.cc:5002] TS bf4709df030f46f8a917017416b009e3 (127.2.197.195:40255): tablet 89ed08d9c43a4cbf93aa1839c67dc16f (table test-table [id=dce8148ae323454bae6c6e9616b4d399]) successfully deleted
W20260429 14:10:01.710952 22222 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.197.193:37821 (error 108)
I20260429 14:10:01.711514 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 22426
I20260429 14:10:01.712656 22693 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:01.712472 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:37821 (local address 127.0.0.1:54236)
0429 14:10:01.712544 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:01.712603 (+ 59us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.193:37821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260429 14:10:01.712845 22222 catalog_manager.cc:4729] TS a0238296c4a14cfc971b48ad960967cf (127.2.197.193:37821): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 89ed08d9c43a4cbf93aa1839c67dc16f: Network error: Client connection negotiation failed: client connection to 127.2.197.193:37821: connect: Connection refused (error 111)
I20260429 14:10:01.717478 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 22557
I20260429 14:10:01.723548 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 22204
2026-04-29T14:10:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (936 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:10:01 dist-test-slave-8n77 krb5kdc[22729](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:10:01 dist-test-slave-8n77 krb5kdc[22729](info): set up 2 sockets
Apr 29 14:10:01 dist-test-slave-8n77 krb5kdc[22729](info): commencing operation
krb5kdc: starting...
W20260429 14:10:03.843922 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.100s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:10:03 dist-test-slave-8n77 krb5kdc[22729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:10:03Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:10:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:10:04.010917 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:40103
--webserver_interface=127.2.197.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:46701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:40103
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:10:04.094148 22745 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:04.094446 22745 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:04.094484 22745 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:10:04.096016 22745 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:10:04.096093 22745 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:04.096117 22745 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:04.096139 22745 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:10:04.096165 22745 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:10:04.097887 22745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:46701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:40103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:40103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:04.098167 22745 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:04.098412 22745 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:10:04.100947 22750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:04.100960 22751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:04.101112 22745 server_base.cc:1061] running on GCE node
W20260429 14:10:04.101146 22753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:04.101567 22745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:04.101821 22745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:04.103005 22745 hybrid_clock.cc:648] HybridClock initialized: now 1777471804102989 us; error 24 us; skew 500 ppm
Apr 29 14:10:04 dist-test-slave-8n77 krb5kdc[22729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471804, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:10:04.105836 22745 init.cc:377] Logged in from keytab as kudu/127.2.197.254@KRBTEST.COM (short username kudu)
I20260429 14:10:04.106434 22745 webserver.cc:492] Webserver started at http://127.2.197.254:33411/ using document root <none> and password file <none>
I20260429 14:10:04.106647 22745 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:04.106689 22745 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:04.106798 22745 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:04.107772 22745 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "a1c57abbade34f1a8221ec20a8ec683e"
format_stamp: "Formatted at 2026-04-29 14:10:04 on dist-test-slave-8n77"
server_key: "d06189114b4c1a7593f90c67e29e5a70"
server_key_iv: "4ef990aba4db4b476338a02a7a7af0d8"
server_key_version: "encryptionkey@0"
I20260429 14:10:04.108114 22745 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "a1c57abbade34f1a8221ec20a8ec683e"
format_stamp: "Formatted at 2026-04-29 14:10:04 on dist-test-slave-8n77"
server_key: "d06189114b4c1a7593f90c67e29e5a70"
server_key_iv: "4ef990aba4db4b476338a02a7a7af0d8"
server_key_version: "encryptionkey@0"
I20260429 14:10:04.109530 22745 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260429 14:10:04.110888 22760 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:04.111366 22745 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:10:04.111446 22745 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "a1c57abbade34f1a8221ec20a8ec683e"
format_stamp: "Formatted at 2026-04-29 14:10:04 on dist-test-slave-8n77"
server_key: "d06189114b4c1a7593f90c67e29e5a70"
server_key_iv: "4ef990aba4db4b476338a02a7a7af0d8"
server_key_version: "encryptionkey@0"
I20260429 14:10:04.111580 22745 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:04.127910 22745 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:04.130543 22745 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:04.130724 22745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:04.135061 22745 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:40103
I20260429 14:10:04.135093 22812 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:40103 every 8 connection(s)
I20260429 14:10:04.135571 22745 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:10:04.136240 22813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:04.138888 22813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1c57abbade34f1a8221ec20a8ec683e: Bootstrap starting.
I20260429 14:10:04.140035 22813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1c57abbade34f1a8221ec20a8ec683e: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:04.140391 22813 log.cc:826] T 00000000000000000000000000000000 P a1c57abbade34f1a8221ec20a8ec683e: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:04.141906 22813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1c57abbade34f1a8221ec20a8ec683e: No bootstrap required, opened a new log
I20260429 14:10:04.143626 22813 raft_consensus.cc:359] T 00000000000000000000000000000000 P a1c57abbade34f1a8221ec20a8ec683e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c57abbade34f1a8221ec20a8ec683e" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40103 } }
I20260429 14:10:04.143800 22813 raft_consensus.cc:385] T 00000000000000000000000000000000 P a1c57abbade34f1a8221ec20a8ec683e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:10:04.143847 22813 raft_consensus.cc:740] T 00000000000000000000000000000000 P a1c57abbade34f1a8221ec20a8ec683e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1c57abbade34f1a8221ec20a8ec683e, State: Initialized, Role: FOLLOWER
I20260429 14:10:04.143986 22813 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1c57abbade34f1a8221ec20a8ec683e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c57abbade34f1a8221ec20a8ec683e" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40103 } }
I20260429 14:10:04.144059 22813 raft_consensus.cc:399] T 00000000000000000000000000000000 P a1c57abbade34f1a8221ec20a8ec683e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:10:04.144101 22813 raft_consensus.cc:493] T 00000000000000000000000000000000 P a1c57abbade34f1a8221ec20a8ec683e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:10:04.144146 22813 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a1c57abbade34f1a8221ec20a8ec683e [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:10:04.144757 22813 raft_consensus.cc:515] T 00000000000000000000000000000000 P a1c57abbade34f1a8221ec20a8ec683e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c57abbade34f1a8221ec20a8ec683e" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40103 } }
I20260429 14:10:04.144891 22813 leader_election.cc:304] T 00000000000000000000000000000000 P a1c57abbade34f1a8221ec20a8ec683e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1c57abbade34f1a8221ec20a8ec683e; no voters:
I20260429 14:10:04.145083 22813 leader_election.cc:290] T 00000000000000000000000000000000 P a1c57abbade34f1a8221ec20a8ec683e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:10:04.145128 22816 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a1c57abbade34f1a8221ec20a8ec683e [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:10:04.145253 22816 raft_consensus.cc:697] T 00000000000000000000000000000000 P a1c57abbade34f1a8221ec20a8ec683e [term 1 LEADER]: Becoming Leader. State: Replica: a1c57abbade34f1a8221ec20a8ec683e, State: Running, Role: LEADER
I20260429 14:10:04.145357 22813 sys_catalog.cc:565] T 00000000000000000000000000000000 P a1c57abbade34f1a8221ec20a8ec683e [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:10:04.145521 22816 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1c57abbade34f1a8221ec20a8ec683e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c57abbade34f1a8221ec20a8ec683e" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40103 } }
I20260429 14:10:04.145838 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 22745
I20260429 14:10:04.145968 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:10:04.146198 2839 external_mini_cluster.cc:1468] Setting key fa4ba33b6166305fb9d3264dc8b4705a
I20260429 14:10:04.146827 22818 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1c57abbade34f1a8221ec20a8ec683e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1c57abbade34f1a8221ec20a8ec683e. Latest consensus state: current_term: 1 leader_uuid: "a1c57abbade34f1a8221ec20a8ec683e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c57abbade34f1a8221ec20a8ec683e" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40103 } } }
I20260429 14:10:04.146930 22818 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1c57abbade34f1a8221ec20a8ec683e [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:04.147149 22817 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1c57abbade34f1a8221ec20a8ec683e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1c57abbade34f1a8221ec20a8ec683e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c57abbade34f1a8221ec20a8ec683e" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 40103 } } }
I20260429 14:10:04.147207 22831 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:10:04.147449 22817 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1c57abbade34f1a8221ec20a8ec683e [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:04.148093 22831 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:10:04.150084 22831 catalog_manager.cc:1357] Generated new cluster ID: 25a8b84972de42e38941115037459ba3
I20260429 14:10:04.150147 22831 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Apr 29 14:10:04 dist-test-slave-8n77 krb5kdc[22729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:10:04.156951 22829 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:04.146886 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42718 (local address 127.2.197.254:40103)
0429 14:10:04.147434 (+ 548us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:04.147446 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:04.147481 (+ 35us) server_negotiation.cc:408] Connection header received
0429 14:10:04.147615 (+ 134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:04.147621 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:04.147684 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:04.148871 (+ 1187us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:04.148971 (+ 100us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:04.149885 (+ 914us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:04.150874 (+ 989us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:04.151118 (+ 244us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:04.153406 (+ 2288us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:04.153428 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:04.153429 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:04.153441 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:04.155472 (+ 2031us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:04.155878 (+ 406us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:04.155881 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:04.155881 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:04.155931 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:04.156106 (+ 175us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:04.156107 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:04.156108 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:04.156311 (+ 203us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:04.156443 (+ 132us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:04.156606 (+ 163us) server_negotiation.cc:300] Negotiation successful
0429 14:10:04.156702 (+ 96us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":79,"threads_started":1}
I20260429 14:10:04.157258 22831 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:10:04.158116 22831 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:10:04.163539 22831 catalog_manager.cc:6044] T 00000000000000000000000000000000 P a1c57abbade34f1a8221ec20a8ec683e: Generated new TSK 0
I20260429 14:10:04.163728 22831 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:10:04.222630 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.193:0
--local_ip_for_outbound_sockets=127.2.197.193
--webserver_interface=127.2.197.193
--webserver_port=0
--tserver_master_addrs=127.2.197.254:40103
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:46701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:10:04.304984 22841 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:04.305174 22841 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:04.305207 22841 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:10:04.306929 22841 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:04.306979 22841 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:04.307009 22841 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.193
I20260429 14:10:04.308849 22841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:46701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.197.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:40103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.193
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:04.309151 22841 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:04.309356 22841 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:10:04.312199 22847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:04.312201 22846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:04.312444 22841 server_base.cc:1061] running on GCE node
W20260429 14:10:04.312199 22849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:04.312786 22841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:04.313020 22841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:04.314209 22841 hybrid_clock.cc:648] HybridClock initialized: now 1777471804314176 us; error 44 us; skew 500 ppm
Apr 29 14:10:04 dist-test-slave-8n77 krb5kdc[22729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471804, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:10:04.317067 22841 init.cc:377] Logged in from keytab as kudu/127.2.197.193@KRBTEST.COM (short username kudu)
I20260429 14:10:04.317670 22841 webserver.cc:492] Webserver started at http://127.2.197.193:46603/ using document root <none> and password file <none>
I20260429 14:10:04.317873 22841 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:04.317934 22841 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:04.318041 22841 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:04.319109 22841 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-0/data/instance:
uuid: "4dd7069f6afc4ffcb324378a734bb56a"
format_stamp: "Formatted at 2026-04-29 14:10:04 on dist-test-slave-8n77"
server_key: "afa530694f7854112572b0b67f21d974"
server_key_iv: "578f2abe224d880d83e4e3bda45527ed"
server_key_version: "encryptionkey@0"
I20260429 14:10:04.319492 22841 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance:
uuid: "4dd7069f6afc4ffcb324378a734bb56a"
format_stamp: "Formatted at 2026-04-29 14:10:04 on dist-test-slave-8n77"
server_key: "afa530694f7854112572b0b67f21d974"
server_key_iv: "578f2abe224d880d83e4e3bda45527ed"
server_key_version: "encryptionkey@0"
I20260429 14:10:04.320832 22841 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260429 14:10:04.321838 22856 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:04.322194 22841 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:10:04.322330 22841 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-0/wal
uuid: "4dd7069f6afc4ffcb324378a734bb56a"
format_stamp: "Formatted at 2026-04-29 14:10:04 on dist-test-slave-8n77"
server_key: "afa530694f7854112572b0b67f21d974"
server_key_iv: "578f2abe224d880d83e4e3bda45527ed"
server_key_version: "encryptionkey@0"
I20260429 14:10:04.322423 22841 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:04.331693 22841 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:04.334517 22841 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:04.334735 22841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:04.335016 22841 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:10:04.335420 22841 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:10:04.335477 22841 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:04.335525 22841 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:10:04.335554 22841 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:04.341740 22841 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.193:36813
I20260429 14:10:04.341807 22969 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.193:36813 every 8 connection(s)
I20260429 14:10:04.342108 22841 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
I20260429 14:10:04.347549 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 22841
I20260429 14:10:04.347709 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance
I20260429 14:10:04.347910 2839 external_mini_cluster.cc:1468] Setting key 858f1a4365527e3b0f589a9c550bf35e
Apr 29 14:10:04 dist-test-slave-8n77 krb5kdc[22729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471804, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.193@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:10:04.351809 22972 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:04.342929 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40103 (local address 127.2.197.193:35577)
0429 14:10:04.343106 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:04.343136 (+ 30us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:04.343644 (+ 508us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:04.343929 (+ 285us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:04.343933 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:04.344249 (+ 316us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:10:04.344524 (+ 275us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:04.344528 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:04.345317 (+ 789us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:04.345318 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:04.345716 (+ 398us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:04.345718 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:04.345822 (+ 104us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:04.347018 (+ 1196us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:10:04.347027 (+ 9us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:10:04.348769 (+ 1742us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:10:04.350593 (+ 1824us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:04.350599 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:04.350601 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:04.350814 (+ 213us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:04.350986 (+ 172us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:04.350987 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:04.350987 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:04.351042 (+ 55us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:04.351352 (+ 310us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:10:04.351355 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:10:04.351461 (+ 106us) client_negotiation.cc:770] Sending connection context
0429 14:10:04.351553 (+ 92us) client_negotiation.cc:241] Negotiation successful
0429 14:10:04.351667 (+ 114us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":92,"threads_started":1}
I20260429 14:10:04.351861 22829 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:04.342971 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:35577 (local address 127.2.197.254:40103)
0429 14:10:04.343057 (+ 86us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:04.343059 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:04.343223 (+ 164us) server_negotiation.cc:408] Connection header received
0429 14:10:04.343766 (+ 543us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:04.343768 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:04.343783 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:04.343842 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:04.344648 (+ 806us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:04.345157 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:04.345848 (+ 691us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:04.345986 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:04.348860 (+ 2874us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:04.348863 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:04.348864 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:04.348872 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:04.350507 (+ 1635us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:04.350893 (+ 386us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:04.350894 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:04.350895 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:04.350921 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:04.351102 (+ 181us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:04.351103 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:04.351104 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:04.351238 (+ 134us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:04.351364 (+ 126us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:04.351600 (+ 236us) server_negotiation.cc:300] Negotiation successful
0429 14:10:04.351717 (+ 117us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260429 14:10:04.352358 22970 heartbeater.cc:344] Connected to a master server at 127.2.197.254:40103
I20260429 14:10:04.352469 22970 heartbeater.cc:461] Registering TS with master...
I20260429 14:10:04.352698 22970 heartbeater.cc:507] Master 127.2.197.254:40103 requested a full tablet report, sending...
I20260429 14:10:04.353281 22777 ts_manager.cc:194] Registered new tserver with Master: 4dd7069f6afc4ffcb324378a734bb56a (127.2.197.193:36813)
I20260429 14:10:04.353996 22777 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.193@KRBTEST.COM'} at 127.2.197.193:35577
WARNING: no policy specified for kudu/127.2.197.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:10:04.402117 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.194:0
--local_ip_for_outbound_sockets=127.2.197.194
--webserver_interface=127.2.197.194
--webserver_port=0
--tserver_master_addrs=127.2.197.254:40103
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:46701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:10:04.480465 22977 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:04.480706 22977 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:04.480751 22977 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:10:04.482398 22977 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:04.482484 22977 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:04.482530 22977 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.194
I20260429 14:10:04.484524 22977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:46701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.197.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:40103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.194
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:04.484817 22977 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:04.485021 22977 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:10:04.488076 22982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:04.488102 22985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:04.488090 22983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:04.489136 22977 server_base.cc:1061] running on GCE node
I20260429 14:10:04.489387 22977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:04.489714 22977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:04.490886 22977 hybrid_clock.cc:648] HybridClock initialized: now 1777471804490875 us; error 34 us; skew 500 ppm
Apr 29 14:10:04 dist-test-slave-8n77 krb5kdc[22729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471804, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:10:04.493750 22977 init.cc:377] Logged in from keytab as kudu/127.2.197.194@KRBTEST.COM (short username kudu)
I20260429 14:10:04.494469 22977 webserver.cc:492] Webserver started at http://127.2.197.194:43425/ using document root <none> and password file <none>
I20260429 14:10:04.494676 22977 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:04.494736 22977 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:04.494853 22977 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:04.495813 22977 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-1/data/instance:
uuid: "bbbd50902be0402b96dbeed20eab914d"
format_stamp: "Formatted at 2026-04-29 14:10:04 on dist-test-slave-8n77"
server_key: "2226be2301d5d75f8a024c546fa14de1"
server_key_iv: "9419ad6d1312264540a729636fb7a508"
server_key_version: "encryptionkey@0"
I20260429 14:10:04.496165 22977 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance:
uuid: "bbbd50902be0402b96dbeed20eab914d"
format_stamp: "Formatted at 2026-04-29 14:10:04 on dist-test-slave-8n77"
server_key: "2226be2301d5d75f8a024c546fa14de1"
server_key_iv: "9419ad6d1312264540a729636fb7a508"
server_key_version: "encryptionkey@0"
I20260429 14:10:04.497673 22977 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260429 14:10:04.498878 22992 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:04.499145 22977 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260429 14:10:04.499248 22977 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-1/wal
uuid: "bbbd50902be0402b96dbeed20eab914d"
format_stamp: "Formatted at 2026-04-29 14:10:04 on dist-test-slave-8n77"
server_key: "2226be2301d5d75f8a024c546fa14de1"
server_key_iv: "9419ad6d1312264540a729636fb7a508"
server_key_version: "encryptionkey@0"
I20260429 14:10:04.499351 22977 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:04.508455 22977 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:04.510545 22977 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:04.510727 22977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:04.510947 22977 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:10:04.511276 22977 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:10:04.511334 22977 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:04.511375 22977 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:10:04.511404 22977 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:04.518049 22977 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.194:32985
I20260429 14:10:04.518101 23105 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.194:32985 every 8 connection(s)
I20260429 14:10:04.518469 22977 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
Apr 29 14:10:04 dist-test-slave-8n77 krb5kdc[22729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471804, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.194@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:10:04.527170 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 22977
I20260429 14:10:04.527324 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance
I20260429 14:10:04.527537 2839 external_mini_cluster.cc:1468] Setting key 080c94092bfffd75a028667e458b67cb
I20260429 14:10:04.530200 22829 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:04.519781 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:50081 (local address 127.2.197.254:40103)
0429 14:10:04.519857 (+ 76us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:04.519860 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:04.520043 (+ 183us) server_negotiation.cc:408] Connection header received
0429 14:10:04.520846 (+ 803us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:04.520847 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:04.520858 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:04.520928 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:04.521855 (+ 927us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:04.522433 (+ 578us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:04.523019 (+ 586us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:04.523143 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:04.525986 (+ 2843us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:04.525992 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:04.525992 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:04.526002 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:04.528404 (+ 2402us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:04.528944 (+ 540us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:04.528945 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:04.528945 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:04.528970 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:04.529384 (+ 414us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:04.529385 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:04.529385 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:04.529484 (+ 99us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:04.529559 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:04.529901 (+ 342us) server_negotiation.cc:300] Negotiation successful
0429 14:10:04.530046 (+ 145us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260429 14:10:04.530330 23108 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:04.519759 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40103 (local address 127.2.197.194:50081)
0429 14:10:04.519927 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:04.519960 (+ 33us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:04.520671 (+ 711us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:04.521055 (+ 384us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:04.521060 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:04.521469 (+ 409us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:10:04.521739 (+ 270us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:04.521743 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:04.522532 (+ 789us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:04.522534 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:04.522938 (+ 404us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:04.522940 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:04.523040 (+ 100us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:04.523973 (+ 933us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:10:04.523991 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:10:04.525873 (+ 1882us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:10:04.528498 (+ 2625us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:04.528507 (+ 9us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:04.528511 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:04.528847 (+ 336us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:04.529049 (+ 202us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:04.529050 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:04.529050 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:04.529149 (+ 99us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:04.529573 (+ 424us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:10:04.529579 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:10:04.529749 (+ 170us) client_negotiation.cc:770] Sending connection context
0429 14:10:04.529874 (+ 125us) client_negotiation.cc:241] Negotiation successful
0429 14:10:04.530042 (+ 168us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":103,"threads_started":1}
I20260429 14:10:04.531101 23106 heartbeater.cc:344] Connected to a master server at 127.2.197.254:40103
I20260429 14:10:04.531186 23106 heartbeater.cc:461] Registering TS with master...
I20260429 14:10:04.531549 23106 heartbeater.cc:507] Master 127.2.197.254:40103 requested a full tablet report, sending...
I20260429 14:10:04.532043 22777 ts_manager.cc:194] Registered new tserver with Master: bbbd50902be0402b96dbeed20eab914d (127.2.197.194:32985)
I20260429 14:10:04.532620 22777 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.194@KRBTEST.COM'} at 127.2.197.194:50081
WARNING: no policy specified for kudu/127.2.197.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:10:04.582706 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.195:0
--local_ip_for_outbound_sockets=127.2.197.195
--webserver_interface=127.2.197.195
--webserver_port=0
--tserver_master_addrs=127.2.197.254:40103
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:46701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:10:04.662122 23113 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:04.662447 23113 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:04.662492 23113 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:10:04.664234 23113 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:04.664319 23113 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:04.664367 23113 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.195
I20260429 14:10:04.666167 23113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:46701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.197.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.197.254:40103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.195
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:04.666569 23113 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:04.666777 23113 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:10:04.669474 23121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:04.669631 23118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:04.669629 23119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:04.669799 23113 server_base.cc:1061] running on GCE node
I20260429 14:10:04.669970 23113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:04.670156 23113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:04.671324 23113 hybrid_clock.cc:648] HybridClock initialized: now 1777471804671322 us; error 47 us; skew 500 ppm
Apr 29 14:10:04 dist-test-slave-8n77 krb5kdc[22729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471804, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:10:04.673940 23113 init.cc:377] Logged in from keytab as kudu/127.2.197.195@KRBTEST.COM (short username kudu)
I20260429 14:10:04.674597 23113 webserver.cc:492] Webserver started at http://127.2.197.195:36887/ using document root <none> and password file <none>
I20260429 14:10:04.674800 23113 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:04.674860 23113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:04.674965 23113 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:04.675972 23113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-2/data/instance:
uuid: "a399a58f19c14eab8e2e6b0b400677fa"
format_stamp: "Formatted at 2026-04-29 14:10:04 on dist-test-slave-8n77"
server_key: "9f6c86fe88a51ff5ae2105993345bc12"
server_key_iv: "ed2b319d0294945fcf3efe462e0a061e"
server_key_version: "encryptionkey@0"
I20260429 14:10:04.676318 23113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance:
uuid: "a399a58f19c14eab8e2e6b0b400677fa"
format_stamp: "Formatted at 2026-04-29 14:10:04 on dist-test-slave-8n77"
server_key: "9f6c86fe88a51ff5ae2105993345bc12"
server_key_iv: "ed2b319d0294945fcf3efe462e0a061e"
server_key_version: "encryptionkey@0"
I20260429 14:10:04.677599 23113 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260429 14:10:04.678496 23128 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:04.678802 23113 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:10:04.678893 23113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-2/wal
uuid: "a399a58f19c14eab8e2e6b0b400677fa"
format_stamp: "Formatted at 2026-04-29 14:10:04 on dist-test-slave-8n77"
server_key: "9f6c86fe88a51ff5ae2105993345bc12"
server_key_iv: "ed2b319d0294945fcf3efe462e0a061e"
server_key_version: "encryptionkey@0"
I20260429 14:10:04.678980 23113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:04.689774 23113 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:04.692022 23113 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:04.692195 23113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:04.692440 23113 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:10:04.692818 23113 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:10:04.692871 23113 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:04.692905 23113 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:10:04.692936 23113 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:04.700541 23113 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.195:45123
I20260429 14:10:04.700613 23241 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.195:45123 every 8 connection(s)
I20260429 14:10:04.701200 23113 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
Apr 29 14:10:04 dist-test-slave-8n77 krb5kdc[22729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471804, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.195@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:10:04.707938 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 23113
I20260429 14:10:04.708081 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance
I20260429 14:10:04.708446 2839 external_mini_cluster.cc:1468] Setting key b546acd4a28f35df840b2fb3196f9638
I20260429 14:10:04.711948 22829 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:04.702763 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:35033 (local address 127.2.197.254:40103)
0429 14:10:04.702854 (+ 91us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:04.702857 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:04.702962 (+ 105us) server_negotiation.cc:408] Connection header received
0429 14:10:04.703618 (+ 656us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:04.703619 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:04.703630 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:04.703688 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:04.704572 (+ 884us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:04.705067 (+ 495us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:04.705741 (+ 674us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:04.705925 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:04.708786 (+ 2861us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:04.708790 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:04.708790 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:04.708800 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:04.710607 (+ 1807us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:04.711073 (+ 466us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:04.711074 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:04.711075 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:04.711104 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:04.711313 (+ 209us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:04.711314 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:04.711315 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:04.711423 (+ 108us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:04.711498 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:04.711762 (+ 264us) server_negotiation.cc:300] Negotiation successful
0429 14:10:04.711859 (+ 97us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260429 14:10:04.712076 23244 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:04.702678 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:40103 (local address 127.2.197.195:35033)
0429 14:10:04.702841 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:04.702875 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:04.703511 (+ 636us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:04.703758 (+ 247us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:04.703762 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:04.704109 (+ 347us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:10:04.704425 (+ 316us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:04.704430 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:04.705166 (+ 736us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:04.705167 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:04.705601 (+ 434us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:04.705603 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:04.705709 (+ 106us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:04.706770 (+ 1061us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:10:04.706780 (+ 10us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:10:04.708674 (+ 1894us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:10:04.710725 (+ 2051us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:04.710732 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:04.710735 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:04.710980 (+ 245us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:04.711170 (+ 190us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:04.711171 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:04.711171 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:04.711245 (+ 74us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:04.711494 (+ 249us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:10:04.711499 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:10:04.711628 (+ 129us) client_negotiation.cc:770] Sending connection context
0429 14:10:04.711742 (+ 114us) client_negotiation.cc:241] Negotiation successful
0429 14:10:04.711883 (+ 141us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":62,"threads_started":1}
I20260429 14:10:04.712750 23242 heartbeater.cc:344] Connected to a master server at 127.2.197.254:40103
I20260429 14:10:04.712850 23242 heartbeater.cc:461] Registering TS with master...
I20260429 14:10:04.713057 23242 heartbeater.cc:507] Master 127.2.197.254:40103 requested a full tablet report, sending...
I20260429 14:10:04.713723 22777 ts_manager.cc:194] Registered new tserver with Master: a399a58f19c14eab8e2e6b0b400677fa (127.2.197.195:45123)
I20260429 14:10:04.714221 22777 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.197.195@KRBTEST.COM'} at 127.2.197.195:35033
I20260429 14:10:04.721022 2839 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260429 14:10:04.728889 22829 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:04.722708 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42726 (local address 127.2.197.254:40103)
0429 14:10:04.722766 (+ 58us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:04.722769 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:04.722780 (+ 11us) server_negotiation.cc:408] Connection header received
0429 14:10:04.722841 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:04.722842 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:04.722854 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:04.722904 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:04.723621 (+ 717us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:04.724120 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:04.724701 (+ 581us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:04.724899 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:04.726196 (+ 1297us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:04.726201 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:04.726201 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:04.726209 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:04.727706 (+ 1497us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:04.728103 (+ 397us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:04.728105 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:04.728106 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:04.728136 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:04.728315 (+ 179us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:04.728315 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:04.728316 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:04.728447 (+ 131us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:04.728567 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:04.728685 (+ 118us) server_negotiation.cc:300] Negotiation successful
0429 14:10:04.728775 (+ 90us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260429 14:10:04.730957 22777 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 14:10:04.732369 22777 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 14:10:04.738683 23253 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:04.735268 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:45123 (local address 127.0.0.1:34858)
0429 14:10:04.735374 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:04.735407 (+ 33us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:04.735478 (+ 71us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:04.736359 (+ 881us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:04.736361 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:04.736364 (+ 3us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:10:04.736604 (+ 240us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:04.736606 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:04.737545 (+ 939us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:04.737546 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:04.738390 (+ 844us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:04.738393 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:04.738463 (+ 70us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:04.738509 (+ 46us) client_negotiation.cc:770] Sending connection context
0429 14:10:04.738559 (+ 50us) client_negotiation.cc:241] Negotiation successful
0429 14:10:04.738600 (+ 41us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":56,"threads_started":1}
I20260429 14:10:04.739045 23256 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:04.735513 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:36813 (local address 127.0.0.1:33836)
0429 14:10:04.735750 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:04.735760 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:04.735805 (+ 45us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:04.736310 (+ 505us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:04.736310 (+ 0us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:04.736312 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:10:04.736522 (+ 210us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:04.736524 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:04.737949 (+ 1425us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:04.737950 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:04.738845 (+ 895us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:04.738847 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:04.738912 (+ 65us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:04.738918 (+ 6us) client_negotiation.cc:770] Sending connection context
0429 14:10:04.738950 (+ 32us) client_negotiation.cc:241] Negotiation successful
0429 14:10:04.738988 (+ 38us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":130,"threads_started":1}
I20260429 14:10:04.739487 23254 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:04.735476 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:32985 (local address 127.0.0.1:35092)
0429 14:10:04.735630 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:04.735640 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:04.735686 (+ 46us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:04.736208 (+ 522us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:04.736210 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:04.736214 (+ 4us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:10:04.736522 (+ 308us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:04.736524 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:04.738347 (+ 1823us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:04.738348 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:04.739263 (+ 915us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:04.739265 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:04.739350 (+ 85us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:04.739356 (+ 6us) client_negotiation.cc:770] Sending connection context
0429 14:10:04.739389 (+ 33us) client_negotiation.cc:241] Negotiation successful
0429 14:10:04.739424 (+ 35us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":110,"threads_started":1}
I20260429 14:10:04.739671 23255 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:04.735463 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34858 (local address 127.2.197.195:45123)
0429 14:10:04.735926 (+ 463us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:04.735936 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:04.735952 (+ 16us) server_negotiation.cc:408] Connection header received
0429 14:10:04.736007 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:04.736009 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:04.736197 (+ 188us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:04.736668 (+ 471us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:10:04.736731 (+ 63us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:04.737442 (+ 711us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:04.738921 (+ 1479us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:04.739450 (+ 529us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:04.739518 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:04.739549 (+ 31us) server_negotiation.cc:300] Negotiation successful
0429 14:10:04.739592 (+ 43us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":130,"threads_started":1}
I20260429 14:10:04.739949 23258 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:04.735934 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33836 (local address 127.2.197.193:36813)
0429 14:10:04.736134 (+ 200us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:04.736142 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:04.736150 (+ 8us) server_negotiation.cc:408] Connection header received
0429 14:10:04.736186 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:04.736188 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:04.736203 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:04.736597 (+ 394us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:10:04.736674 (+ 77us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:04.737860 (+ 1186us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:04.739249 (+ 1389us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:04.739744 (+ 495us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:04.739786 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:04.739816 (+ 30us) server_negotiation.cc:300] Negotiation successful
0429 14:10:04.739855 (+ 39us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":113,"threads_started":1}
I20260429 14:10:04.740259 23257 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:04.735782 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35092 (local address 127.2.197.194:32985)
0429 14:10:04.736011 (+ 229us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:04.736020 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:04.736030 (+ 10us) server_negotiation.cc:408] Connection header received
0429 14:10:04.736071 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:04.736073 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:04.736091 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:04.736605 (+ 514us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:10:04.736673 (+ 68us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:04.737860 (+ 1187us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:04.739370 (+ 1510us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:04.740081 (+ 711us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:04.740119 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:04.740148 (+ 29us) server_negotiation.cc:300] Negotiation successful
0429 14:10:04.740185 (+ 37us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":112,"threads_started":1}
I20260429 14:10:04.740252 23176 tablet_service.cc:1511] Processing CreateTablet for tablet f585c072958747f4b655e11b68538637 (DEFAULT_TABLE table=test-table [id=193dc291fa4b4e8e91971ee923b97ebc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:10:04.740434 22903 tablet_service.cc:1511] Processing CreateTablet for tablet f585c072958747f4b655e11b68538637 (DEFAULT_TABLE table=test-table [id=193dc291fa4b4e8e91971ee923b97ebc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:10:04.740572 23176 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f585c072958747f4b655e11b68538637. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:04.740705 22903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f585c072958747f4b655e11b68538637. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:04.740774 23040 tablet_service.cc:1511] Processing CreateTablet for tablet f585c072958747f4b655e11b68538637 (DEFAULT_TABLE table=test-table [id=193dc291fa4b4e8e91971ee923b97ebc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:10:04.740979 23040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f585c072958747f4b655e11b68538637. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:04.742868 23259 tablet_bootstrap.cc:492] T f585c072958747f4b655e11b68538637 P 4dd7069f6afc4ffcb324378a734bb56a: Bootstrap starting.
I20260429 14:10:04.743651 23260 tablet_bootstrap.cc:492] T f585c072958747f4b655e11b68538637 P a399a58f19c14eab8e2e6b0b400677fa: Bootstrap starting.
I20260429 14:10:04.743804 23259 tablet_bootstrap.cc:654] T f585c072958747f4b655e11b68538637 P 4dd7069f6afc4ffcb324378a734bb56a: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:04.744187 23259 log.cc:826] T f585c072958747f4b655e11b68538637 P 4dd7069f6afc4ffcb324378a734bb56a: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:04.744262 23260 tablet_bootstrap.cc:654] T f585c072958747f4b655e11b68538637 P a399a58f19c14eab8e2e6b0b400677fa: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:04.744333 23261 tablet_bootstrap.cc:492] T f585c072958747f4b655e11b68538637 P bbbd50902be0402b96dbeed20eab914d: Bootstrap starting.
I20260429 14:10:04.744531 23260 log.cc:826] T f585c072958747f4b655e11b68538637 P a399a58f19c14eab8e2e6b0b400677fa: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:04.744908 23261 tablet_bootstrap.cc:654] T f585c072958747f4b655e11b68538637 P bbbd50902be0402b96dbeed20eab914d: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:04.745059 23259 tablet_bootstrap.cc:492] T f585c072958747f4b655e11b68538637 P 4dd7069f6afc4ffcb324378a734bb56a: No bootstrap required, opened a new log
I20260429 14:10:04.745112 23259 ts_tablet_manager.cc:1403] T f585c072958747f4b655e11b68538637 P 4dd7069f6afc4ffcb324378a734bb56a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260429 14:10:04.745329 23261 log.cc:826] T f585c072958747f4b655e11b68538637 P bbbd50902be0402b96dbeed20eab914d: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:04.746560 23261 tablet_bootstrap.cc:492] T f585c072958747f4b655e11b68538637 P bbbd50902be0402b96dbeed20eab914d: No bootstrap required, opened a new log
I20260429 14:10:04.746649 23261 ts_tablet_manager.cc:1403] T f585c072958747f4b655e11b68538637 P bbbd50902be0402b96dbeed20eab914d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260429 14:10:04.746691 23260 tablet_bootstrap.cc:492] T f585c072958747f4b655e11b68538637 P a399a58f19c14eab8e2e6b0b400677fa: No bootstrap required, opened a new log
I20260429 14:10:04.746755 23260 ts_tablet_manager.cc:1403] T f585c072958747f4b655e11b68538637 P a399a58f19c14eab8e2e6b0b400677fa: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260429 14:10:04.746997 23259 raft_consensus.cc:359] T f585c072958747f4b655e11b68538637 P 4dd7069f6afc4ffcb324378a734bb56a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd7069f6afc4ffcb324378a734bb56a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 36813 } } peers { permanent_uuid: "a399a58f19c14eab8e2e6b0b400677fa" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 45123 } } peers { permanent_uuid: "bbbd50902be0402b96dbeed20eab914d" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 32985 } }
I20260429 14:10:04.747160 23259 raft_consensus.cc:385] T f585c072958747f4b655e11b68538637 P 4dd7069f6afc4ffcb324378a734bb56a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:10:04.747195 23259 raft_consensus.cc:740] T f585c072958747f4b655e11b68538637 P 4dd7069f6afc4ffcb324378a734bb56a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dd7069f6afc4ffcb324378a734bb56a, State: Initialized, Role: FOLLOWER
I20260429 14:10:04.747334 23259 consensus_queue.cc:260] T f585c072958747f4b655e11b68538637 P 4dd7069f6afc4ffcb324378a734bb56a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd7069f6afc4ffcb324378a734bb56a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 36813 } } peers { permanent_uuid: "a399a58f19c14eab8e2e6b0b400677fa" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 45123 } } peers { permanent_uuid: "bbbd50902be0402b96dbeed20eab914d" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 32985 } }
I20260429 14:10:04.747601 23259 ts_tablet_manager.cc:1434] T f585c072958747f4b655e11b68538637 P 4dd7069f6afc4ffcb324378a734bb56a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260429 14:10:04.747742 22970 heartbeater.cc:499] Master 127.2.197.254:40103 was elected leader, sending a full tablet report...
I20260429 14:10:04.748456 23260 raft_consensus.cc:359] T f585c072958747f4b655e11b68538637 P a399a58f19c14eab8e2e6b0b400677fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd7069f6afc4ffcb324378a734bb56a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 36813 } } peers { permanent_uuid: "a399a58f19c14eab8e2e6b0b400677fa" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 45123 } } peers { permanent_uuid: "bbbd50902be0402b96dbeed20eab914d" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 32985 } }
I20260429 14:10:04.748624 23260 raft_consensus.cc:385] T f585c072958747f4b655e11b68538637 P a399a58f19c14eab8e2e6b0b400677fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:10:04.748663 23260 raft_consensus.cc:740] T f585c072958747f4b655e11b68538637 P a399a58f19c14eab8e2e6b0b400677fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a399a58f19c14eab8e2e6b0b400677fa, State: Initialized, Role: FOLLOWER
I20260429 14:10:04.748935 23261 raft_consensus.cc:359] T f585c072958747f4b655e11b68538637 P bbbd50902be0402b96dbeed20eab914d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd7069f6afc4ffcb324378a734bb56a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 36813 } } peers { permanent_uuid: "a399a58f19c14eab8e2e6b0b400677fa" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 45123 } } peers { permanent_uuid: "bbbd50902be0402b96dbeed20eab914d" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 32985 } }
I20260429 14:10:04.748766 23260 consensus_queue.cc:260] T f585c072958747f4b655e11b68538637 P a399a58f19c14eab8e2e6b0b400677fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd7069f6afc4ffcb324378a734bb56a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 36813 } } peers { permanent_uuid: "a399a58f19c14eab8e2e6b0b400677fa" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 45123 } } peers { permanent_uuid: "bbbd50902be0402b96dbeed20eab914d" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 32985 } }
I20260429 14:10:04.749081 23261 raft_consensus.cc:385] T f585c072958747f4b655e11b68538637 P bbbd50902be0402b96dbeed20eab914d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:10:04.749115 23261 raft_consensus.cc:740] T f585c072958747f4b655e11b68538637 P bbbd50902be0402b96dbeed20eab914d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbbd50902be0402b96dbeed20eab914d, State: Initialized, Role: FOLLOWER
I20260429 14:10:04.749198 23261 consensus_queue.cc:260] T f585c072958747f4b655e11b68538637 P bbbd50902be0402b96dbeed20eab914d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd7069f6afc4ffcb324378a734bb56a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 36813 } } peers { permanent_uuid: "a399a58f19c14eab8e2e6b0b400677fa" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 45123 } } peers { permanent_uuid: "bbbd50902be0402b96dbeed20eab914d" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 32985 } }
I20260429 14:10:04.749327 23260 ts_tablet_manager.cc:1434] T f585c072958747f4b655e11b68538637 P a399a58f19c14eab8e2e6b0b400677fa: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260429 14:10:04.749400 23242 heartbeater.cc:499] Master 127.2.197.254:40103 was elected leader, sending a full tablet report...
I20260429 14:10:04.749465 23261 ts_tablet_manager.cc:1434] T f585c072958747f4b655e11b68538637 P bbbd50902be0402b96dbeed20eab914d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260429 14:10:04.749562 23106 heartbeater.cc:499] Master 127.2.197.254:40103 was elected leader, sending a full tablet report...
W20260429 14:10:04.769058 23107 tablet.cc:2404] T f585c072958747f4b655e11b68538637 P bbbd50902be0402b96dbeed20eab914d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260429 14:10:04.842978 22971 tablet.cc:2404] T f585c072958747f4b655e11b68538637 P 4dd7069f6afc4ffcb324378a734bb56a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260429 14:10:04.952150 23243 tablet.cc:2404] T f585c072958747f4b655e11b68538637 P a399a58f19c14eab8e2e6b0b400677fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 14:10:05.000857 23265 raft_consensus.cc:493] T f585c072958747f4b655e11b68538637 P 4dd7069f6afc4ffcb324378a734bb56a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:10:05.000995 23265 raft_consensus.cc:515] T f585c072958747f4b655e11b68538637 P 4dd7069f6afc4ffcb324378a734bb56a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd7069f6afc4ffcb324378a734bb56a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 36813 } } peers { permanent_uuid: "a399a58f19c14eab8e2e6b0b400677fa" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 45123 } } peers { permanent_uuid: "bbbd50902be0402b96dbeed20eab914d" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 32985 } }
I20260429 14:10:05.001513 23265 leader_election.cc:290] T f585c072958747f4b655e11b68538637 P 4dd7069f6afc4ffcb324378a734bb56a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a399a58f19c14eab8e2e6b0b400677fa (127.2.197.195:45123), bbbd50902be0402b96dbeed20eab914d (127.2.197.194:32985)
I20260429 14:10:05.005108 23268 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:05.001769 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:45123 (local address 127.2.197.193:57833)
0429 14:10:05.001902 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:05.001915 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:05.001979 (+ 64us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:05.002519 (+ 540us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:05.002520 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:05.002523 (+ 3us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:10:05.002799 (+ 276us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:05.002802 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.003875 (+ 1073us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:05.003876 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:05.004804 (+ 928us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:05.004807 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.004922 (+ 115us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:05.004927 (+ 5us) client_negotiation.cc:770] Sending connection context
0429 14:10:05.004959 (+ 32us) client_negotiation.cc:241] Negotiation successful
0429 14:10:05.004999 (+ 40us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260429 14:10:05.005209 23269 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:05.002022 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:32985 (local address 127.2.197.193:56933)
0429 14:10:05.002231 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:05.002245 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:05.002353 (+ 108us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:05.002739 (+ 386us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:05.002741 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:05.002745 (+ 4us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:10:05.003039 (+ 294us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:05.003041 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.004161 (+ 1120us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:05.004163 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:05.005000 (+ 837us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:05.005002 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.005074 (+ 72us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:05.005076 (+ 2us) client_negotiation.cc:770] Sending connection context
0429 14:10:05.005107 (+ 31us) client_negotiation.cc:241] Negotiation successful
0429 14:10:05.005143 (+ 36us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":41,"threads_started":1}
I20260429 14:10:05.005726 23255 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:05.002313 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:57833 (local address 127.2.197.195:45123)
0429 14:10:05.002383 (+ 70us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:05.002385 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:05.002395 (+ 10us) server_negotiation.cc:408] Connection header received
0429 14:10:05.002428 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:05.002429 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:05.002437 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:05.002509 (+ 72us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:10:05.002904 (+ 395us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.003710 (+ 806us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:05.004949 (+ 1239us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.005577 (+ 628us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:05.005588 (+ 11us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:05.005617 (+ 29us) server_negotiation.cc:300] Negotiation successful
0429 14:10:05.005655 (+ 38us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260429 14:10:05.005821 23257 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:05.002179 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:56933 (local address 127.2.197.194:32985)
0429 14:10:05.002247 (+ 68us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:05.002249 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:05.002559 (+ 310us) server_negotiation.cc:408] Connection header received
0429 14:10:05.002587 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:05.002588 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:05.002600 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:05.002658 (+ 58us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:10:05.003241 (+ 583us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.004041 (+ 800us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:05.005126 (+ 1085us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.005676 (+ 550us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:05.005685 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:05.005709 (+ 24us) server_negotiation.cc:300] Negotiation successful
0429 14:10:05.005750 (+ 41us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260429 14:10:05.006218 23060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f585c072958747f4b655e11b68538637" candidate_uuid: "4dd7069f6afc4ffcb324378a734bb56a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbbd50902be0402b96dbeed20eab914d" is_pre_election: true
I20260429 14:10:05.006455 23060 raft_consensus.cc:2468] T f585c072958747f4b655e11b68538637 P bbbd50902be0402b96dbeed20eab914d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dd7069f6afc4ffcb324378a734bb56a in term 0.
I20260429 14:10:05.006250 23196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f585c072958747f4b655e11b68538637" candidate_uuid: "4dd7069f6afc4ffcb324378a734bb56a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a399a58f19c14eab8e2e6b0b400677fa" is_pre_election: true
I20260429 14:10:05.006711 23196 raft_consensus.cc:2468] T f585c072958747f4b655e11b68538637 P a399a58f19c14eab8e2e6b0b400677fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dd7069f6afc4ffcb324378a734bb56a in term 0.
I20260429 14:10:05.006932 22858 leader_election.cc:304] T f585c072958747f4b655e11b68538637 P 4dd7069f6afc4ffcb324378a734bb56a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dd7069f6afc4ffcb324378a734bb56a, bbbd50902be0402b96dbeed20eab914d; no voters:
I20260429 14:10:05.007158 23265 raft_consensus.cc:2804] T f585c072958747f4b655e11b68538637 P 4dd7069f6afc4ffcb324378a734bb56a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 14:10:05.007238 23265 raft_consensus.cc:493] T f585c072958747f4b655e11b68538637 P 4dd7069f6afc4ffcb324378a734bb56a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 14:10:05.007458 23265 raft_consensus.cc:3060] T f585c072958747f4b655e11b68538637 P 4dd7069f6afc4ffcb324378a734bb56a [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:10:05.008410 23265 raft_consensus.cc:515] T f585c072958747f4b655e11b68538637 P 4dd7069f6afc4ffcb324378a734bb56a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd7069f6afc4ffcb324378a734bb56a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 36813 } } peers { permanent_uuid: "a399a58f19c14eab8e2e6b0b400677fa" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 45123 } } peers { permanent_uuid: "bbbd50902be0402b96dbeed20eab914d" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 32985 } }
I20260429 14:10:05.008562 23265 leader_election.cc:290] T f585c072958747f4b655e11b68538637 P 4dd7069f6afc4ffcb324378a734bb56a [CANDIDATE]: Term 1 election: Requested vote from peers a399a58f19c14eab8e2e6b0b400677fa (127.2.197.195:45123), bbbd50902be0402b96dbeed20eab914d (127.2.197.194:32985)
I20260429 14:10:05.008875 23060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f585c072958747f4b655e11b68538637" candidate_uuid: "4dd7069f6afc4ffcb324378a734bb56a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbbd50902be0402b96dbeed20eab914d"
I20260429 14:10:05.008986 23060 raft_consensus.cc:3060] T f585c072958747f4b655e11b68538637 P bbbd50902be0402b96dbeed20eab914d [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:10:05.008822 23196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f585c072958747f4b655e11b68538637" candidate_uuid: "4dd7069f6afc4ffcb324378a734bb56a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a399a58f19c14eab8e2e6b0b400677fa"
I20260429 14:10:05.009099 23196 raft_consensus.cc:3060] T f585c072958747f4b655e11b68538637 P a399a58f19c14eab8e2e6b0b400677fa [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:10:05.009920 23060 raft_consensus.cc:2468] T f585c072958747f4b655e11b68538637 P bbbd50902be0402b96dbeed20eab914d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dd7069f6afc4ffcb324378a734bb56a in term 1.
I20260429 14:10:05.009960 23196 raft_consensus.cc:2468] T f585c072958747f4b655e11b68538637 P a399a58f19c14eab8e2e6b0b400677fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dd7069f6afc4ffcb324378a734bb56a in term 1.
I20260429 14:10:05.010242 22858 leader_election.cc:304] T f585c072958747f4b655e11b68538637 P 4dd7069f6afc4ffcb324378a734bb56a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dd7069f6afc4ffcb324378a734bb56a, a399a58f19c14eab8e2e6b0b400677fa; no voters:
I20260429 14:10:05.010443 23265 raft_consensus.cc:2804] T f585c072958747f4b655e11b68538637 P 4dd7069f6afc4ffcb324378a734bb56a [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:10:05.010632 23265 raft_consensus.cc:697] T f585c072958747f4b655e11b68538637 P 4dd7069f6afc4ffcb324378a734bb56a [term 1 LEADER]: Becoming Leader. State: Replica: 4dd7069f6afc4ffcb324378a734bb56a, State: Running, Role: LEADER
I20260429 14:10:05.010723 23265 consensus_queue.cc:237] T f585c072958747f4b655e11b68538637 P 4dd7069f6afc4ffcb324378a734bb56a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd7069f6afc4ffcb324378a734bb56a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 36813 } } peers { permanent_uuid: "a399a58f19c14eab8e2e6b0b400677fa" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 45123 } } peers { permanent_uuid: "bbbd50902be0402b96dbeed20eab914d" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 32985 } }
I20260429 14:10:05.011751 22776 catalog_manager.cc:5671] T f585c072958747f4b655e11b68538637 P 4dd7069f6afc4ffcb324378a734bb56a reported cstate change: term changed from 0 to 1, leader changed from <none> to 4dd7069f6afc4ffcb324378a734bb56a (127.2.197.193). New cstate: current_term: 1 leader_uuid: "4dd7069f6afc4ffcb324378a734bb56a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd7069f6afc4ffcb324378a734bb56a" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 36813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a399a58f19c14eab8e2e6b0b400677fa" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 45123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbbd50902be0402b96dbeed20eab914d" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 32985 } health_report { overall_health: UNKNOWN } } }
I20260429 14:10:05.030536 23258 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:05.028325 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33838 (local address 127.2.197.193:36813)
0429 14:10:05.028401 (+ 76us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:05.028403 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:05.028412 (+ 9us) server_negotiation.cc:408] Connection header received
0429 14:10:05.028464 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:05.028465 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:05.028480 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:05.028557 (+ 77us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:10:05.028823 (+ 266us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.029342 (+ 519us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:05.029940 (+ 598us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.030095 (+ 155us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:05.030125 (+ 30us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:10:05.030216 (+ 91us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:10:05.030339 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:05.030397 (+ 58us) server_negotiation.cc:300] Negotiation successful
0429 14:10:05.030422 (+ 25us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260429 14:10:05.032296 23196 raft_consensus.cc:1275] T f585c072958747f4b655e11b68538637 P a399a58f19c14eab8e2e6b0b400677fa [term 1 FOLLOWER]: Refusing update from remote peer 4dd7069f6afc4ffcb324378a734bb56a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:10:05.032686 23060 raft_consensus.cc:1275] T f585c072958747f4b655e11b68538637 P bbbd50902be0402b96dbeed20eab914d [term 1 FOLLOWER]: Refusing update from remote peer 4dd7069f6afc4ffcb324378a734bb56a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:10:05.032779 23265 consensus_queue.cc:1048] T f585c072958747f4b655e11b68538637 P 4dd7069f6afc4ffcb324378a734bb56a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a399a58f19c14eab8e2e6b0b400677fa" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 45123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:10:05.033089 23265 consensus_queue.cc:1048] T f585c072958747f4b655e11b68538637 P 4dd7069f6afc4ffcb324378a734bb56a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbbd50902be0402b96dbeed20eab914d" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 32985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:10:05.035842 23274 mvcc.cc:204] Tried to move back new op lower bound from 7280524513409134592 to 7280524513324756992. Current Snapshot: MvccSnapshot[applied={T|T < 7280524513409134592}]
I20260429 14:10:05.037758 22777 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:10:05.037842 22777 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:10:05.038679 22777 catalog_manager.cc:5958] T 00000000000000000000000000000000 P a1c57abbade34f1a8221ec20a8ec683e: Sending DeleteTablet for 3 replicas of tablet f585c072958747f4b655e11b68538637
I20260429 14:10:05.039099 23176 tablet_service.cc:1558] Processing DeleteTablet for tablet f585c072958747f4b655e11b68538637 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:10:05 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:34858
I20260429 14:10:05.039358 23040 tablet_service.cc:1558] Processing DeleteTablet for tablet f585c072958747f4b655e11b68538637 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:10:05 UTC) from {username='kudu', principal='kudu/127.2.197.254@KRBTEST.COM'} at 127.0.0.1:35092
I20260429 14:10:05.039386 23282 tablet_replica.cc:333] T f585c072958747f4b655e11b68538637 P a399a58f19c14eab8e2e6b0b400677fa: stopping tablet replica
I20260429 14:10:05.039501 23282 raft_consensus.cc:2243] T f585c072958747f4b655e11b68538637 P a399a58f19c14eab8e2e6b0b400677fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:10:05.039605 23282 raft_consensus.cc:2272] T f585c072958747f4b655e11b68538637 P a399a58f19c14eab8e2e6b0b400677fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:10:05.039695 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 22841
I20260429 14:10:05.041404 23282 ts_tablet_manager.cc:1916] T f585c072958747f4b655e11b68538637 P a399a58f19c14eab8e2e6b0b400677fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:10:05.041551 23283 tablet_replica.cc:333] T f585c072958747f4b655e11b68538637 P bbbd50902be0402b96dbeed20eab914d: stopping tablet replica
I20260429 14:10:05.041654 23283 raft_consensus.cc:2243] T f585c072958747f4b655e11b68538637 P bbbd50902be0402b96dbeed20eab914d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:10:05.041757 23283 raft_consensus.cc:2272] T f585c072958747f4b655e11b68538637 P bbbd50902be0402b96dbeed20eab914d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:10:05.043615 23282 ts_tablet_manager.cc:1929] T f585c072958747f4b655e11b68538637 P a399a58f19c14eab8e2e6b0b400677fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 14:10:05.043685 23282 log.cc:1199] T f585c072958747f4b655e11b68538637 P a399a58f19c14eab8e2e6b0b400677fa: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777471413505459-2839-0/minicluster-data/ts-2/wal/wals/f585c072958747f4b655e11b68538637
I20260429 14:10:05.043944 23282 ts_tablet_manager.cc:1950] T f585c072958747f4b655e11b68538637 P a399a58f19c14eab8e2e6b0b400677fa: Deleting consensus metadata
I20260429 14:10:05.044405 22762 catalog_manager.cc:5002] TS a399a58f19c14eab8e2e6b0b400677fa (127.2.197.195:45123): tablet f585c072958747f4b655e11b68538637 (table test-table [id=193dc291fa4b4e8e91971ee923b97ebc]) successfully deleted
W20260429 14:10:05.047933 22764 connection.cc:570] client connection to 127.2.197.193:36813 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260429 14:10:05.048024 22764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260429 14:10:05.048245 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 22977
I20260429 14:10:05.048925 23254 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:05.048413 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:36813 (local address 127.0.0.1:33852)
0429 14:10:05.048853 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:05.048894 (+ 41us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.193:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418}
W20260429 14:10:05.049042 22764 catalog_manager.cc:4729] TS 4dd7069f6afc4ffcb324378a734bb56a (127.2.197.193:36813): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f585c072958747f4b655e11b68538637: Network error: Client connection negotiation failed: client connection to 127.2.197.193:36813: connect: Connection refused (error 111)
I20260429 14:10:05.055271 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 23113
I20260429 14:10:05.055608 23254 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:05.055288 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:32985 (local address 127.0.0.1:35104)
0429 14:10:05.055464 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:05.055537 (+ 73us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.194:32985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260429 14:10:05.061285 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 22745
2026-04-29T14:10:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (3339 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-04-29T14:10:05Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:10:05Z Disabled control of system clock
I20260429 14:10:05.091473 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:45683
--webserver_interface=127.2.197.254
--webserver_port=0
--builtin_ntp_servers=127.2.197.212:41851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:45683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260429 14:10:05.171620 23288 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:05.171833 23288 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:05.171857 23288 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:10:05.173631 23288 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:10:05.173693 23288 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:05.173707 23288 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:05.173719 23288 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:10:05.173732 23288 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:10:05.175655 23288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:41851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:45683
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:45683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:05.175926 23288 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:05.176201 23288 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:10:05.178722 23296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:05.178788 23293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:05.178885 23288 server_base.cc:1061] running on GCE node
W20260429 14:10:05.178711 23294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:05.179168 23288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:05.179412 23288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:05.180649 23288 hybrid_clock.cc:648] HybridClock initialized: now 1777471805180636 us; error 48 us; skew 500 ppm
I20260429 14:10:05.182000 23288 webserver.cc:492] Webserver started at http://127.2.197.254:44395/ using document root <none> and password file <none>
I20260429 14:10:05.182230 23288 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:05.182473 23288 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:05.183012 23288 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:05.184531 23288 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "4acc652221a54445b515de7a245526d2"
format_stamp: "Formatted at 2026-04-29 14:10:05 on dist-test-slave-8n77"
server_key: "17a9fdc251e6f2f2509c96bd728c787f"
server_key_iv: "1ca2560b5340828250bf8a98b9ca747c"
server_key_version: "encryptionkey@0"
I20260429 14:10:05.184886 23288 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "4acc652221a54445b515de7a245526d2"
format_stamp: "Formatted at 2026-04-29 14:10:05 on dist-test-slave-8n77"
server_key: "17a9fdc251e6f2f2509c96bd728c787f"
server_key_iv: "1ca2560b5340828250bf8a98b9ca747c"
server_key_version: "encryptionkey@0"
I20260429 14:10:05.187480 23288 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.005s sys 0.000s
I20260429 14:10:05.188745 23302 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:05.188998 23288 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:10:05.189078 23288 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "4acc652221a54445b515de7a245526d2"
format_stamp: "Formatted at 2026-04-29 14:10:05 on dist-test-slave-8n77"
server_key: "17a9fdc251e6f2f2509c96bd728c787f"
server_key_iv: "1ca2560b5340828250bf8a98b9ca747c"
server_key_version: "encryptionkey@0"
I20260429 14:10:05.189172 23288 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:05.198599 23288 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:05.198860 23288 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:05.198954 23288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:05.202803 23288 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:45683
I20260429 14:10:05.202875 23354 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:45683 every 8 connection(s)
I20260429 14:10:05.203176 23288 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:10:05.203781 23355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:05.206118 23355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4acc652221a54445b515de7a245526d2: Bootstrap starting.
I20260429 14:10:05.206133 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 23288
I20260429 14:10:05.206317 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:10:05.206534 2839 external_mini_cluster.cc:1468] Setting key 3d83d7e87bccd8d87ab6bc9758a65255
I20260429 14:10:05.206835 23355 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4acc652221a54445b515de7a245526d2: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:05.207195 23355 log.cc:826] T 00000000000000000000000000000000 P 4acc652221a54445b515de7a245526d2: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:05.208106 23355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4acc652221a54445b515de7a245526d2: No bootstrap required, opened a new log
I20260429 14:10:05.209416 23355 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4acc652221a54445b515de7a245526d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4acc652221a54445b515de7a245526d2" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 45683 } }
I20260429 14:10:05.209574 23355 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4acc652221a54445b515de7a245526d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:10:05.209611 23355 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4acc652221a54445b515de7a245526d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4acc652221a54445b515de7a245526d2, State: Initialized, Role: FOLLOWER
I20260429 14:10:05.209714 23355 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4acc652221a54445b515de7a245526d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4acc652221a54445b515de7a245526d2" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 45683 } }
I20260429 14:10:05.209791 23355 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4acc652221a54445b515de7a245526d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:10:05.209834 23355 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4acc652221a54445b515de7a245526d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:10:05.209892 23355 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4acc652221a54445b515de7a245526d2 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:10:05.210667 23355 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4acc652221a54445b515de7a245526d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4acc652221a54445b515de7a245526d2" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 45683 } }
I20260429 14:10:05.210765 23355 leader_election.cc:304] T 00000000000000000000000000000000 P 4acc652221a54445b515de7a245526d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4acc652221a54445b515de7a245526d2; no voters:
I20260429 14:10:05.210922 23355 leader_election.cc:290] T 00000000000000000000000000000000 P 4acc652221a54445b515de7a245526d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:10:05.210963 23360 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4acc652221a54445b515de7a245526d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:10:05.211072 23360 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4acc652221a54445b515de7a245526d2 [term 1 LEADER]: Becoming Leader. State: Replica: 4acc652221a54445b515de7a245526d2, State: Running, Role: LEADER
I20260429 14:10:05.211163 23360 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4acc652221a54445b515de7a245526d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4acc652221a54445b515de7a245526d2" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 45683 } }
I20260429 14:10:05.211186 23355 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4acc652221a54445b515de7a245526d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:10:05.211501 23361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4acc652221a54445b515de7a245526d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4acc652221a54445b515de7a245526d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4acc652221a54445b515de7a245526d2" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 45683 } } }
I20260429 14:10:05.211501 23362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4acc652221a54445b515de7a245526d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4acc652221a54445b515de7a245526d2. Latest consensus state: current_term: 1 leader_uuid: "4acc652221a54445b515de7a245526d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4acc652221a54445b515de7a245526d2" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 45683 } } }
I20260429 14:10:05.211628 23361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4acc652221a54445b515de7a245526d2 [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:05.211633 23362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4acc652221a54445b515de7a245526d2 [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:05.211817 23364 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:10:05.212517 23364 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:10:05.214123 23364 catalog_manager.cc:1357] Generated new cluster ID: 06e83304b2664d03be8dd1042a162689
I20260429 14:10:05.214178 23364 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:10:05.216739 23358 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:05.207084 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:39014 (local address 127.2.197.254:45683)
0429 14:10:05.207289 (+ 205us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:05.207296 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:05.207320 (+ 24us) server_negotiation.cc:408] Connection header received
0429 14:10:05.207652 (+ 332us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:05.207658 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:05.207701 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:05.207801 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:05.208323 (+ 522us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.209037 (+ 714us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:05.209910 (+ 873us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.210157 (+ 247us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:05.216121 (+ 5964us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:05.216129 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:05.216129 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:10:05.216336 (+ 207us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:05.216357 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:10:05.216365 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:05.216445 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:05.216554 (+ 109us) server_negotiation.cc:300] Negotiation successful
0429 14:10:05.216611 (+ 57us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":103,"threads_started":1}
I20260429 14:10:05.223615 23364 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:10:05.224406 23364 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:10:05.233872 23364 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 4acc652221a54445b515de7a245526d2: Generated new TSK 0
I20260429 14:10:05.234113 23364 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 14:10:05.242072 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.193:0
--local_ip_for_outbound_sockets=127.2.197.193
--webserver_interface=127.2.197.193
--webserver_port=0
--tserver_master_addrs=127.2.197.254:45683
--builtin_ntp_servers=127.2.197.212:41851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260429 14:10:05.326229 23379 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:05.326532 23379 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:05.326572 23379 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:10:05.328433 23379 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:05.328505 23379 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:05.328552 23379 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.193
I20260429 14:10:05.330219 23379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:41851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.197.193
--webserver_port=0
--tserver_master_addrs=127.2.197.254:45683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.193
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:05.330513 23379 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:05.330771 23379 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:10:05.333581 23384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:05.333604 23387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:05.333581 23385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:05.334728 23379 server_base.cc:1061] running on GCE node
I20260429 14:10:05.335026 23379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:05.335330 23379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:05.336510 23379 hybrid_clock.cc:648] HybridClock initialized: now 1777471805336486 us; error 40 us; skew 500 ppm
I20260429 14:10:05.337865 23379 webserver.cc:492] Webserver started at http://127.2.197.193:39023/ using document root <none> and password file <none>
I20260429 14:10:05.338069 23379 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:05.338136 23379 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:05.338240 23379 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:05.339224 23379 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-0/data/instance:
uuid: "d4e3309ca61a466a93b6563e4e9ed705"
format_stamp: "Formatted at 2026-04-29 14:10:05 on dist-test-slave-8n77"
server_key: "ebb883a1a58473f9182ca95033182e5d"
server_key_iv: "c26600c3a0df4261185c475e4808ce04"
server_key_version: "encryptionkey@0"
I20260429 14:10:05.339591 23379 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance:
uuid: "d4e3309ca61a466a93b6563e4e9ed705"
format_stamp: "Formatted at 2026-04-29 14:10:05 on dist-test-slave-8n77"
server_key: "ebb883a1a58473f9182ca95033182e5d"
server_key_iv: "c26600c3a0df4261185c475e4808ce04"
server_key_version: "encryptionkey@0"
I20260429 14:10:05.340794 23379 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260429 14:10:05.341566 23393 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:05.341814 23379 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:10:05.341900 23379 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-0/wal
uuid: "d4e3309ca61a466a93b6563e4e9ed705"
format_stamp: "Formatted at 2026-04-29 14:10:05 on dist-test-slave-8n77"
server_key: "ebb883a1a58473f9182ca95033182e5d"
server_key_iv: "c26600c3a0df4261185c475e4808ce04"
server_key_version: "encryptionkey@0"
I20260429 14:10:05.341971 23379 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:05.361981 23379 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:05.362353 23379 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:05.362509 23379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:05.362783 23379 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:10:05.363168 23379 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:10:05.363221 23379 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:05.363255 23379 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:10:05.363272 23379 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:05.370353 23379 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.193:39357
I20260429 14:10:05.370518 23506 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.193:39357 every 8 connection(s)
I20260429 14:10:05.370923 23379 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-0/data/info.pb
I20260429 14:10:05.375381 23358 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:05.371772 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.193:42985 (local address 127.2.197.254:45683)
0429 14:10:05.371851 (+ 79us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:05.371854 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:05.372042 (+ 188us) server_negotiation.cc:408] Connection header received
0429 14:10:05.372622 (+ 580us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:05.372624 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:05.372637 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:05.372697 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:05.373145 (+ 448us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.373669 (+ 524us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:05.374300 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.374487 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:05.375070 (+ 583us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:05.375073 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:05.375074 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:10:05.375079 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:05.375093 (+ 14us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:10:05.375097 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:05.375168 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:05.375269 (+ 101us) server_negotiation.cc:300] Negotiation successful
0429 14:10:05.375304 (+ 35us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260429 14:10:05.375566 23509 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:05.371714 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:45683 (local address 127.2.197.193:42985)
0429 14:10:05.371917 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:05.371952 (+ 35us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:05.372515 (+ 563us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:05.372770 (+ 255us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:05.372774 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:05.372782 (+ 8us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:10:05.373053 (+ 271us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:05.373055 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.373775 (+ 720us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:05.373776 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:05.374193 (+ 417us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:05.374195 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.374355 (+ 160us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:05.374955 (+ 600us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 14:10:05.374955 (+ 0us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:10:05.374967 (+ 12us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 14:10:05.374986 (+ 19us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:10:05.375168 (+ 182us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:10:05.375173 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:10:05.375176 (+ 3us) client_negotiation.cc:770] Sending connection context
0429 14:10:05.375275 (+ 99us) client_negotiation.cc:241] Negotiation successful
0429 14:10:05.375344 (+ 69us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":72,"threads_started":1}
I20260429 14:10:05.376066 23507 heartbeater.cc:344] Connected to a master server at 127.2.197.254:45683
I20260429 14:10:05.376152 23507 heartbeater.cc:461] Registering TS with master...
I20260429 14:10:05.376353 23507 heartbeater.cc:507] Master 127.2.197.254:45683 requested a full tablet report, sending...
I20260429 14:10:05.376849 23315 ts_manager.cc:194] Registered new tserver with Master: d4e3309ca61a466a93b6563e4e9ed705 (127.2.197.193:39357)
I20260429 14:10:05.377632 23315 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.2.197.193:42985
I20260429 14:10:05.377883 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 23379
I20260429 14:10:05.377988 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-0/wal/instance
I20260429 14:10:05.378189 2839 external_mini_cluster.cc:1468] Setting key c192a98b8fae59d33206837a19320477
I20260429 14:10:05.379125 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.194:0
--local_ip_for_outbound_sockets=127.2.197.194
--webserver_interface=127.2.197.194
--webserver_port=0
--tserver_master_addrs=127.2.197.254:45683
--builtin_ntp_servers=127.2.197.212:41851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260429 14:10:05.456863 23510 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:05.457046 23510 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:05.457067 23510 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:10:05.458634 23510 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:05.458685 23510 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:05.458711 23510 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.194
I20260429 14:10:05.460477 23510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:41851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.197.194
--webserver_port=0
--tserver_master_addrs=127.2.197.254:45683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.194
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:05.460705 23510 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:05.460891 23510 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:10:05.463783 23516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:05.463764 23518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:05.463919 23510 server_base.cc:1061] running on GCE node
W20260429 14:10:05.463764 23515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:05.464195 23510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:05.464444 23510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:05.465615 23510 hybrid_clock.cc:648] HybridClock initialized: now 1777471805465594 us; error 43 us; skew 500 ppm
I20260429 14:10:05.467012 23510 webserver.cc:492] Webserver started at http://127.2.197.194:38957/ using document root <none> and password file <none>
I20260429 14:10:05.467231 23510 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:05.467320 23510 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:05.467439 23510 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:05.468387 23510 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-1/data/instance:
uuid: "42f5c7503a8f4ee8be4b4b58752aab9e"
format_stamp: "Formatted at 2026-04-29 14:10:05 on dist-test-slave-8n77"
server_key: "27d3b39fd744b10bc7c29d3ed72ee1c2"
server_key_iv: "a33af015af2e3ebd900ad04ed8e19a41"
server_key_version: "encryptionkey@0"
I20260429 14:10:05.468726 23510 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance:
uuid: "42f5c7503a8f4ee8be4b4b58752aab9e"
format_stamp: "Formatted at 2026-04-29 14:10:05 on dist-test-slave-8n77"
server_key: "27d3b39fd744b10bc7c29d3ed72ee1c2"
server_key_iv: "a33af015af2e3ebd900ad04ed8e19a41"
server_key_version: "encryptionkey@0"
I20260429 14:10:05.469928 23510 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260429 14:10:05.470773 23524 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:05.470952 23510 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:05.471031 23510 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-1/wal
uuid: "42f5c7503a8f4ee8be4b4b58752aab9e"
format_stamp: "Formatted at 2026-04-29 14:10:05 on dist-test-slave-8n77"
server_key: "27d3b39fd744b10bc7c29d3ed72ee1c2"
server_key_iv: "a33af015af2e3ebd900ad04ed8e19a41"
server_key_version: "encryptionkey@0"
I20260429 14:10:05.471112 23510 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:05.486253 23510 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:05.486672 23510 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:05.486810 23510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:05.487031 23510 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:10:05.487366 23510 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:10:05.487423 23510 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:05.487468 23510 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:10:05.487493 23510 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:05.493127 23510 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.194:38083
I20260429 14:10:05.493188 23637 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.194:38083 every 8 connection(s)
I20260429 14:10:05.493528 23510 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-1/data/info.pb
I20260429 14:10:05.497753 23358 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:05.494224 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.194:47443 (local address 127.2.197.254:45683)
0429 14:10:05.494333 (+ 109us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:05.494335 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:05.494546 (+ 211us) server_negotiation.cc:408] Connection header received
0429 14:10:05.495066 (+ 520us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:05.495067 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:05.495077 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:05.495125 (+ 48us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:05.495693 (+ 568us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.496176 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:05.496859 (+ 683us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.496990 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:05.497505 (+ 515us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:05.497508 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:05.497509 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:10:05.497513 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:05.497526 (+ 13us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:10:05.497529 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:05.497578 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:05.497665 (+ 87us) server_negotiation.cc:300] Negotiation successful
0429 14:10:05.497694 (+ 29us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260429 14:10:05.497849 23640 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:05.494240 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:45683 (local address 127.2.197.194:47443)
0429 14:10:05.494439 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:05.494465 (+ 26us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:05.494972 (+ 507us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:05.495187 (+ 215us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:05.495191 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:05.495201 (+ 10us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:10:05.495571 (+ 370us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:05.495576 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.496297 (+ 721us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:05.496299 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:05.496747 (+ 448us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:05.496750 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.496857 (+ 107us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:05.497383 (+ 526us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 14:10:05.497384 (+ 1us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:10:05.497395 (+ 11us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 14:10:05.497416 (+ 21us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:10:05.497592 (+ 176us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:10:05.497595 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:10:05.497597 (+ 2us) client_negotiation.cc:770] Sending connection context
0429 14:10:05.497652 (+ 55us) client_negotiation.cc:241] Negotiation successful
0429 14:10:05.497726 (+ 74us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":98,"threads_started":1}
I20260429 14:10:05.498566 23638 heartbeater.cc:344] Connected to a master server at 127.2.197.254:45683
I20260429 14:10:05.498692 23638 heartbeater.cc:461] Registering TS with master...
I20260429 14:10:05.498907 23638 heartbeater.cc:507] Master 127.2.197.254:45683 requested a full tablet report, sending...
I20260429 14:10:05.499419 23315 ts_manager.cc:194] Registered new tserver with Master: 42f5c7503a8f4ee8be4b4b58752aab9e (127.2.197.194:38083)
I20260429 14:10:05.499970 23315 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.2.197.194:47443
I20260429 14:10:05.503758 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 23510
I20260429 14:10:05.503858 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-1/wal/instance
I20260429 14:10:05.504004 2839 external_mini_cluster.cc:1468] Setting key 0df999b5fd6e9b21ede8b714fd04cbe8
I20260429 14:10:05.505018 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.197.195:0
--local_ip_for_outbound_sockets=127.2.197.195
--webserver_interface=127.2.197.195
--webserver_port=0
--tserver_master_addrs=127.2.197.254:45683
--builtin_ntp_servers=127.2.197.212:41851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260429 14:10:05.587459 23641 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:05.587647 23641 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:05.587670 23641 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:10:05.589243 23641 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:05.589315 23641 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:05.589344 23641 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.197.195
I20260429 14:10:05.590994 23641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:41851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.197.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.197.195
--webserver_port=0
--tserver_master_addrs=127.2.197.254:45683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.197.195
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:05.591207 23641 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:05.591400 23641 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:10:05.594167 23646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:05.594607 23641 server_base.cc:1061] running on GCE node
W20260429 14:10:05.594255 23647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:05.594172 23649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:05.594888 23641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:05.595122 23641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:05.596313 23641 hybrid_clock.cc:648] HybridClock initialized: now 1777471805596277 us; error 69 us; skew 500 ppm
I20260429 14:10:05.597540 23641 webserver.cc:492] Webserver started at http://127.2.197.195:38627/ using document root <none> and password file <none>
I20260429 14:10:05.597754 23641 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:05.597816 23641 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:05.597919 23641 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:05.598981 23641 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-2/data/instance:
uuid: "96695e33efb845dd97f83b3c703f8ee6"
format_stamp: "Formatted at 2026-04-29 14:10:05 on dist-test-slave-8n77"
server_key: "583026c8683ded35bf4c7439b45e613e"
server_key_iv: "24ed16d9c20b44810f7804126c137b5d"
server_key_version: "encryptionkey@0"
I20260429 14:10:05.599340 23641 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance:
uuid: "96695e33efb845dd97f83b3c703f8ee6"
format_stamp: "Formatted at 2026-04-29 14:10:05 on dist-test-slave-8n77"
server_key: "583026c8683ded35bf4c7439b45e613e"
server_key_iv: "24ed16d9c20b44810f7804126c137b5d"
server_key_version: "encryptionkey@0"
I20260429 14:10:05.600559 23641 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260429 14:10:05.601397 23655 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:05.601666 23641 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:10:05.601748 23641 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-2/wal
uuid: "96695e33efb845dd97f83b3c703f8ee6"
format_stamp: "Formatted at 2026-04-29 14:10:05 on dist-test-slave-8n77"
server_key: "583026c8683ded35bf4c7439b45e613e"
server_key_iv: "24ed16d9c20b44810f7804126c137b5d"
server_key_version: "encryptionkey@0"
I20260429 14:10:05.601830 23641 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:05.637982 23641 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:05.638381 23641 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:05.638542 23641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:05.638792 23641 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 14:10:05.639137 23641 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 14:10:05.639191 23641 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:05.639227 23641 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 14:10:05.639283 23641 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:05.645484 23641 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.195:41271
I20260429 14:10:05.645541 23768 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.195:41271 every 8 connection(s)
I20260429 14:10:05.645834 23641 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-2/data/info.pb
I20260429 14:10:05.650041 23358 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:05.646667 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:37011 (local address 127.2.197.254:45683)
0429 14:10:05.646739 (+ 72us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:05.646742 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:05.646996 (+ 254us) server_negotiation.cc:408] Connection header received
0429 14:10:05.647465 (+ 469us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:05.647466 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:05.647477 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:05.647558 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:05.648019 (+ 461us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.648539 (+ 520us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:05.649099 (+ 560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.649234 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:05.649761 (+ 527us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:05.649765 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:05.649765 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:10:05.649769 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:05.649783 (+ 14us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:10:05.649786 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:05.649838 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:05.649961 (+ 123us) server_negotiation.cc:300] Negotiation successful
0429 14:10:05.649986 (+ 25us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260429 14:10:05.650125 23771 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:05.646673 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:45683 (local address 127.2.197.195:37011)
0429 14:10:05.646891 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:05.646918 (+ 27us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:05.647385 (+ 467us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:05.647639 (+ 254us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:05.647642 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:05.647650 (+ 8us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:10:05.647920 (+ 270us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:05.647922 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.648632 (+ 710us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:05.648633 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:05.649006 (+ 373us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:05.649008 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.649103 (+ 95us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:05.649644 (+ 541us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 14:10:05.649645 (+ 1us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:10:05.649656 (+ 11us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 14:10:05.649675 (+ 19us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:10:05.649872 (+ 197us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:10:05.649877 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:10:05.649880 (+ 3us) client_negotiation.cc:770] Sending connection context
0429 14:10:05.649953 (+ 73us) client_negotiation.cc:241] Negotiation successful
0429 14:10:05.650008 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":94,"threads_started":1}
I20260429 14:10:05.650460 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 23641
I20260429 14:10:05.650601 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-2/wal/instance
I20260429 14:10:05.650751 23769 heartbeater.cc:344] Connected to a master server at 127.2.197.254:45683
I20260429 14:10:05.650795 2839 external_mini_cluster.cc:1468] Setting key 721a0ce24217c71f95665e139e744b14
I20260429 14:10:05.650839 23769 heartbeater.cc:461] Registering TS with master...
I20260429 14:10:05.651047 23769 heartbeater.cc:507] Master 127.2.197.254:45683 requested a full tablet report, sending...
I20260429 14:10:05.651407 23315 ts_manager.cc:194] Registered new tserver with Master: 96695e33efb845dd97f83b3c703f8ee6 (127.2.197.195:41271)
I20260429 14:10:05.651831 2839 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260429 14:10:05.651911 23315 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.2.197.195:37011
I20260429 14:10:05.655941 23358 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:05.653302 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:39022 (local address 127.2.197.254:45683)
0429 14:10:05.653345 (+ 43us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:05.653351 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:05.653460 (+ 109us) server_negotiation.cc:408] Connection header received
0429 14:10:05.653485 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:05.653486 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:05.653508 (+ 22us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:05.653576 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:05.653939 (+ 363us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.654495 (+ 556us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:05.655059 (+ 564us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.655176 (+ 117us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:05.655691 (+ 515us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:05.655694 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:05.655695 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:10:05.655698 (+ 3us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:05.655711 (+ 13us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:10:05.655714 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:05.655765 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:05.655865 (+ 100us) server_negotiation.cc:300] Negotiation successful
0429 14:10:05.655888 (+ 23us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260429 14:10:05.657616 23315 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39022:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 14:10:05.659080 23315 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 14:10:05.664916 23778 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:05.661426 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:39357 (local address 127.0.0.1:36430)
0429 14:10:05.661705 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:05.661715 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:05.661773 (+ 58us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:05.662049 (+ 276us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:05.662052 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:05.662057 (+ 5us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:10:05.662354 (+ 297us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:05.662357 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.663602 (+ 1245us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:05.663604 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:05.664605 (+ 1001us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:05.664607 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.664689 (+ 82us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:05.664733 (+ 44us) client_negotiation.cc:770] Sending connection context
0429 14:10:05.664782 (+ 49us) client_negotiation.cc:241] Negotiation successful
0429 14:10:05.664830 (+ 48us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":110,"threads_started":1}
I20260429 14:10:05.665413 23781 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:05.661600 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:38083 (local address 127.0.0.1:48072)
0429 14:10:05.661858 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:05.661869 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:05.661927 (+ 58us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:05.662173 (+ 246us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:05.662174 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:05.662176 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:10:05.662406 (+ 230us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:05.662409 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.664321 (+ 1912us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:05.664322 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:05.665223 (+ 901us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:05.665225 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.665306 (+ 81us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:05.665310 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 14:10:05.665334 (+ 24us) client_negotiation.cc:241] Negotiation successful
0429 14:10:05.665364 (+ 30us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":106,"threads_started":1}
I20260429 14:10:05.665431 23780 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:05.661541 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36430 (local address 127.2.197.193:39357)
0429 14:10:05.661736 (+ 195us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:05.661745 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:05.661757 (+ 12us) server_negotiation.cc:408] Connection header received
0429 14:10:05.661870 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:05.661873 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:05.661892 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:05.661953 (+ 61us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:10:05.662446 (+ 493us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.663508 (+ 1062us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:05.664688 (+ 1180us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.665195 (+ 507us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:05.665265 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:05.665321 (+ 56us) server_negotiation.cc:300] Negotiation successful
0429 14:10:05.665356 (+ 35us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":94,"threads_started":1}
I20260429 14:10:05.665932 23441 tablet_service.cc:1511] Processing CreateTablet for tablet 6d36de0ebbf64e77b71ac9605f83d2c4 (DEFAULT_TABLE table=test-table [id=3be43436782c4da7b598623cd4e55683]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:10:05.666097 23777 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:05.661373 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.195:41271 (local address 127.0.0.1:56632)
0429 14:10:05.661619 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:05.661652 (+ 33us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:05.661733 (+ 81us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:05.662049 (+ 316us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:05.662052 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:05.662055 (+ 3us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:10:05.662592 (+ 537us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:05.662595 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.664381 (+ 1786us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:05.664381 (+ 0us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:05.665870 (+ 1489us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:05.665873 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.665949 (+ 76us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:05.665953 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 14:10:05.665985 (+ 32us) client_negotiation.cc:241] Negotiation successful
0429 14:10:05.666026 (+ 41us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":80,"threads_started":1}
I20260429 14:10:05.666265 23441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d36de0ebbf64e77b71ac9605f83d2c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:05.666759 23779 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:05.661488 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:56632 (local address 127.2.197.195:41271)
0429 14:10:05.661852 (+ 364us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:05.661860 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:05.661869 (+ 9us) server_negotiation.cc:408] Connection header received
0429 14:10:05.661902 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:05.661904 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:05.661919 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:05.661976 (+ 57us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:10:05.662712 (+ 736us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.664295 (+ 1583us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:05.665962 (+ 1667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.666531 (+ 569us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:05.666613 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:05.666645 (+ 32us) server_negotiation.cc:300] Negotiation successful
0429 14:10:05.666684 (+ 39us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":78,"threads_started":1}
I20260429 14:10:05.666989 23782 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:05.661857 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48072 (local address 127.2.197.194:38083)
0429 14:10:05.662016 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:05.662022 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:05.662028 (+ 6us) server_negotiation.cc:408] Connection header received
0429 14:10:05.662055 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:05.662056 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:05.662068 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:05.662147 (+ 79us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:10:05.662498 (+ 351us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.664200 (+ 1702us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:05.665325 (+ 1125us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.666621 (+ 1296us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:05.666679 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:05.666707 (+ 28us) server_negotiation.cc:300] Negotiation successful
0429 14:10:05.666744 (+ 37us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":81,"threads_started":1}
I20260429 14:10:05.667321 23703 tablet_service.cc:1511] Processing CreateTablet for tablet 6d36de0ebbf64e77b71ac9605f83d2c4 (DEFAULT_TABLE table=test-table [id=3be43436782c4da7b598623cd4e55683]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:10:05.667595 23703 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d36de0ebbf64e77b71ac9605f83d2c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:05.667593 23572 tablet_service.cc:1511] Processing CreateTablet for tablet 6d36de0ebbf64e77b71ac9605f83d2c4 (DEFAULT_TABLE table=test-table [id=3be43436782c4da7b598623cd4e55683]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 14:10:05.667840 23572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d36de0ebbf64e77b71ac9605f83d2c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:05.669301 23783 tablet_bootstrap.cc:492] T 6d36de0ebbf64e77b71ac9605f83d2c4 P d4e3309ca61a466a93b6563e4e9ed705: Bootstrap starting.
I20260429 14:10:05.669991 23783 tablet_bootstrap.cc:654] T 6d36de0ebbf64e77b71ac9605f83d2c4 P d4e3309ca61a466a93b6563e4e9ed705: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:05.670104 23784 tablet_bootstrap.cc:492] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 42f5c7503a8f4ee8be4b4b58752aab9e: Bootstrap starting.
I20260429 14:10:05.670547 23785 tablet_bootstrap.cc:492] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 96695e33efb845dd97f83b3c703f8ee6: Bootstrap starting.
I20260429 14:10:05.670572 23783 log.cc:826] T 6d36de0ebbf64e77b71ac9605f83d2c4 P d4e3309ca61a466a93b6563e4e9ed705: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:05.670734 23784 tablet_bootstrap.cc:654] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 42f5c7503a8f4ee8be4b4b58752aab9e: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:05.671026 23784 log.cc:826] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 42f5c7503a8f4ee8be4b4b58752aab9e: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:05.671119 23785 tablet_bootstrap.cc:654] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 96695e33efb845dd97f83b3c703f8ee6: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:05.671411 23785 log.cc:826] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 96695e33efb845dd97f83b3c703f8ee6: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:05.671509 23783 tablet_bootstrap.cc:492] T 6d36de0ebbf64e77b71ac9605f83d2c4 P d4e3309ca61a466a93b6563e4e9ed705: No bootstrap required, opened a new log
I20260429 14:10:05.671593 23783 ts_tablet_manager.cc:1403] T 6d36de0ebbf64e77b71ac9605f83d2c4 P d4e3309ca61a466a93b6563e4e9ed705: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260429 14:10:05.671897 23784 tablet_bootstrap.cc:492] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 42f5c7503a8f4ee8be4b4b58752aab9e: No bootstrap required, opened a new log
I20260429 14:10:05.671952 23784 ts_tablet_manager.cc:1403] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 42f5c7503a8f4ee8be4b4b58752aab9e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260429 14:10:05.672299 23785 tablet_bootstrap.cc:492] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 96695e33efb845dd97f83b3c703f8ee6: No bootstrap required, opened a new log
I20260429 14:10:05.672410 23785 ts_tablet_manager.cc:1403] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 96695e33efb845dd97f83b3c703f8ee6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260429 14:10:05.673034 23784 raft_consensus.cc:359] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 42f5c7503a8f4ee8be4b4b58752aab9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4e3309ca61a466a93b6563e4e9ed705" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 39357 } } peers { permanent_uuid: "96695e33efb845dd97f83b3c703f8ee6" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 41271 } } peers { permanent_uuid: "42f5c7503a8f4ee8be4b4b58752aab9e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38083 } }
I20260429 14:10:05.673151 23784 raft_consensus.cc:385] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 42f5c7503a8f4ee8be4b4b58752aab9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:10:05.673173 23784 raft_consensus.cc:740] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 42f5c7503a8f4ee8be4b4b58752aab9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42f5c7503a8f4ee8be4b4b58752aab9e, State: Initialized, Role: FOLLOWER
I20260429 14:10:05.673295 23784 consensus_queue.cc:260] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 42f5c7503a8f4ee8be4b4b58752aab9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4e3309ca61a466a93b6563e4e9ed705" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 39357 } } peers { permanent_uuid: "96695e33efb845dd97f83b3c703f8ee6" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 41271 } } peers { permanent_uuid: "42f5c7503a8f4ee8be4b4b58752aab9e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38083 } }
I20260429 14:10:05.673408 23783 raft_consensus.cc:359] T 6d36de0ebbf64e77b71ac9605f83d2c4 P d4e3309ca61a466a93b6563e4e9ed705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4e3309ca61a466a93b6563e4e9ed705" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 39357 } } peers { permanent_uuid: "96695e33efb845dd97f83b3c703f8ee6" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 41271 } } peers { permanent_uuid: "42f5c7503a8f4ee8be4b4b58752aab9e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38083 } }
I20260429 14:10:05.673566 23783 raft_consensus.cc:385] T 6d36de0ebbf64e77b71ac9605f83d2c4 P d4e3309ca61a466a93b6563e4e9ed705 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:10:05.673547 23784 ts_tablet_manager.cc:1434] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 42f5c7503a8f4ee8be4b4b58752aab9e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260429 14:10:05.673599 23783 raft_consensus.cc:740] T 6d36de0ebbf64e77b71ac9605f83d2c4 P d4e3309ca61a466a93b6563e4e9ed705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4e3309ca61a466a93b6563e4e9ed705, State: Initialized, Role: FOLLOWER
I20260429 14:10:05.673679 23638 heartbeater.cc:499] Master 127.2.197.254:45683 was elected leader, sending a full tablet report...
I20260429 14:10:05.673710 23783 consensus_queue.cc:260] T 6d36de0ebbf64e77b71ac9605f83d2c4 P d4e3309ca61a466a93b6563e4e9ed705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4e3309ca61a466a93b6563e4e9ed705" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 39357 } } peers { permanent_uuid: "96695e33efb845dd97f83b3c703f8ee6" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 41271 } } peers { permanent_uuid: "42f5c7503a8f4ee8be4b4b58752aab9e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38083 } }
I20260429 14:10:05.673928 23783 ts_tablet_manager.cc:1434] T 6d36de0ebbf64e77b71ac9605f83d2c4 P d4e3309ca61a466a93b6563e4e9ed705: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260429 14:10:05.674045 23507 heartbeater.cc:499] Master 127.2.197.254:45683 was elected leader, sending a full tablet report...
I20260429 14:10:05.674237 23785 raft_consensus.cc:359] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 96695e33efb845dd97f83b3c703f8ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4e3309ca61a466a93b6563e4e9ed705" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 39357 } } peers { permanent_uuid: "96695e33efb845dd97f83b3c703f8ee6" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 41271 } } peers { permanent_uuid: "42f5c7503a8f4ee8be4b4b58752aab9e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38083 } }
I20260429 14:10:05.674412 23785 raft_consensus.cc:385] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 96695e33efb845dd97f83b3c703f8ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:10:05.674434 23785 raft_consensus.cc:740] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 96695e33efb845dd97f83b3c703f8ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96695e33efb845dd97f83b3c703f8ee6, State: Initialized, Role: FOLLOWER
I20260429 14:10:05.674517 23785 consensus_queue.cc:260] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 96695e33efb845dd97f83b3c703f8ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4e3309ca61a466a93b6563e4e9ed705" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 39357 } } peers { permanent_uuid: "96695e33efb845dd97f83b3c703f8ee6" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 41271 } } peers { permanent_uuid: "42f5c7503a8f4ee8be4b4b58752aab9e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38083 } }
I20260429 14:10:05.674749 23785 ts_tablet_manager.cc:1434] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 96695e33efb845dd97f83b3c703f8ee6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260429 14:10:05.674808 23769 heartbeater.cc:499] Master 127.2.197.254:45683 was elected leader, sending a full tablet report...
I20260429 14:10:05.740937 23791 raft_consensus.cc:493] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 96695e33efb845dd97f83b3c703f8ee6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 14:10:05.741065 23791 raft_consensus.cc:515] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 96695e33efb845dd97f83b3c703f8ee6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4e3309ca61a466a93b6563e4e9ed705" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 39357 } } peers { permanent_uuid: "96695e33efb845dd97f83b3c703f8ee6" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 41271 } } peers { permanent_uuid: "42f5c7503a8f4ee8be4b4b58752aab9e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38083 } }
I20260429 14:10:05.741629 23791 leader_election.cc:290] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 96695e33efb845dd97f83b3c703f8ee6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4e3309ca61a466a93b6563e4e9ed705 (127.2.197.193:39357), 42f5c7503a8f4ee8be4b4b58752aab9e (127.2.197.194:38083)
W20260429 14:10:05.744038 23639 tablet.cc:2404] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 42f5c7503a8f4ee8be4b4b58752aab9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 14:10:05.744306 23771 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:05.741860 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.194:38083 (local address 127.2.197.195:35767)
0429 14:10:05.741934 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:05.741949 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:05.742019 (+ 70us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:05.742210 (+ 191us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:05.742211 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:05.742213 (+ 2us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:10:05.742572 (+ 359us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:05.742574 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.743454 (+ 880us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:05.743455 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:05.744057 (+ 602us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:05.744058 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.744157 (+ 99us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:05.744160 (+ 3us) client_negotiation.cc:770] Sending connection context
0429 14:10:05.744194 (+ 34us) client_negotiation.cc:241] Negotiation successful
0429 14:10:05.744227 (+ 33us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260429 14:10:05.744741 23792 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:05.741865 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:39357 (local address 127.2.197.195:47963)
0429 14:10:05.742199 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:05.742210 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:05.742372 (+ 162us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:05.742509 (+ 137us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:05.742510 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:05.742511 (+ 1us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 14:10:05.742974 (+ 463us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:05.742977 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.743690 (+ 713us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:05.743691 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:05.744459 (+ 768us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:05.744462 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.744582 (+ 120us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:05.744587 (+ 5us) client_negotiation.cc:770] Sending connection context
0429 14:10:05.744622 (+ 35us) client_negotiation.cc:241] Negotiation successful
0429 14:10:05.744660 (+ 38us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":78,"thread_start_us":143,"threads_started":1}
I20260429 14:10:05.745172 23782 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:05.742014 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:35767 (local address 127.2.197.194:38083)
0429 14:10:05.742060 (+ 46us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:05.742063 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:05.742073 (+ 10us) server_negotiation.cc:408] Connection header received
0429 14:10:05.742104 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:05.742105 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:05.742117 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:05.742178 (+ 61us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:10:05.742693 (+ 515us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.743190 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:05.744154 (+ 964us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.745071 (+ 917us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:05.745079 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:05.745103 (+ 24us) server_negotiation.cc:300] Negotiation successful
0429 14:10:05.745128 (+ 25us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260429 14:10:05.745389 23780 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:05.741977 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.2.197.195:47963 (local address 127.2.197.193:39357)
0429 14:10:05.742025 (+ 48us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:05.742028 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:05.742382 (+ 354us) server_negotiation.cc:408] Connection header received
0429 14:10:05.742423 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:05.742424 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:05.742435 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:05.742490 (+ 55us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 14:10:05.743087 (+ 597us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.743591 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:05.744604 (+ 1013us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.745172 (+ 568us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:05.745180 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:05.745242 (+ 62us) server_negotiation.cc:300] Negotiation successful
0429 14:10:05.745321 (+ 79us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260429 14:10:05.745564 23592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d36de0ebbf64e77b71ac9605f83d2c4" candidate_uuid: "96695e33efb845dd97f83b3c703f8ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42f5c7503a8f4ee8be4b4b58752aab9e" is_pre_election: true
I20260429 14:10:05.745754 23592 raft_consensus.cc:2468] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 42f5c7503a8f4ee8be4b4b58752aab9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96695e33efb845dd97f83b3c703f8ee6 in term 0.
I20260429 14:10:05.745951 23461 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d36de0ebbf64e77b71ac9605f83d2c4" candidate_uuid: "96695e33efb845dd97f83b3c703f8ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4e3309ca61a466a93b6563e4e9ed705" is_pre_election: true
I20260429 14:10:05.746007 23659 leader_election.cc:304] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 96695e33efb845dd97f83b3c703f8ee6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42f5c7503a8f4ee8be4b4b58752aab9e, 96695e33efb845dd97f83b3c703f8ee6; no voters:
I20260429 14:10:05.746122 23461 raft_consensus.cc:2468] T 6d36de0ebbf64e77b71ac9605f83d2c4 P d4e3309ca61a466a93b6563e4e9ed705 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96695e33efb845dd97f83b3c703f8ee6 in term 0.
I20260429 14:10:05.746151 23791 raft_consensus.cc:2804] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 96695e33efb845dd97f83b3c703f8ee6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 14:10:05.746192 23791 raft_consensus.cc:493] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 96695e33efb845dd97f83b3c703f8ee6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 14:10:05.746214 23791 raft_consensus.cc:3060] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 96695e33efb845dd97f83b3c703f8ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:10:05.747089 23791 raft_consensus.cc:515] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 96695e33efb845dd97f83b3c703f8ee6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4e3309ca61a466a93b6563e4e9ed705" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 39357 } } peers { permanent_uuid: "96695e33efb845dd97f83b3c703f8ee6" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 41271 } } peers { permanent_uuid: "42f5c7503a8f4ee8be4b4b58752aab9e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38083 } }
I20260429 14:10:05.747244 23791 leader_election.cc:290] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 96695e33efb845dd97f83b3c703f8ee6 [CANDIDATE]: Term 1 election: Requested vote from peers d4e3309ca61a466a93b6563e4e9ed705 (127.2.197.193:39357), 42f5c7503a8f4ee8be4b4b58752aab9e (127.2.197.194:38083)
I20260429 14:10:05.747417 23461 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d36de0ebbf64e77b71ac9605f83d2c4" candidate_uuid: "96695e33efb845dd97f83b3c703f8ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4e3309ca61a466a93b6563e4e9ed705"
I20260429 14:10:05.747483 23592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d36de0ebbf64e77b71ac9605f83d2c4" candidate_uuid: "96695e33efb845dd97f83b3c703f8ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42f5c7503a8f4ee8be4b4b58752aab9e"
I20260429 14:10:05.747506 23461 raft_consensus.cc:3060] T 6d36de0ebbf64e77b71ac9605f83d2c4 P d4e3309ca61a466a93b6563e4e9ed705 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:10:05.747551 23592 raft_consensus.cc:3060] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 42f5c7503a8f4ee8be4b4b58752aab9e [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:10:05.748255 23592 raft_consensus.cc:2468] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 42f5c7503a8f4ee8be4b4b58752aab9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96695e33efb845dd97f83b3c703f8ee6 in term 1.
I20260429 14:10:05.748258 23461 raft_consensus.cc:2468] T 6d36de0ebbf64e77b71ac9605f83d2c4 P d4e3309ca61a466a93b6563e4e9ed705 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96695e33efb845dd97f83b3c703f8ee6 in term 1.
I20260429 14:10:05.748442 23659 leader_election.cc:304] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 96695e33efb845dd97f83b3c703f8ee6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42f5c7503a8f4ee8be4b4b58752aab9e, 96695e33efb845dd97f83b3c703f8ee6; no voters:
I20260429 14:10:05.748574 23791 raft_consensus.cc:2804] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 96695e33efb845dd97f83b3c703f8ee6 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:10:05.748745 23791 raft_consensus.cc:697] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 96695e33efb845dd97f83b3c703f8ee6 [term 1 LEADER]: Becoming Leader. State: Replica: 96695e33efb845dd97f83b3c703f8ee6, State: Running, Role: LEADER
I20260429 14:10:05.748858 23791 consensus_queue.cc:237] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 96695e33efb845dd97f83b3c703f8ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4e3309ca61a466a93b6563e4e9ed705" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 39357 } } peers { permanent_uuid: "96695e33efb845dd97f83b3c703f8ee6" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 41271 } } peers { permanent_uuid: "42f5c7503a8f4ee8be4b4b58752aab9e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38083 } }
I20260429 14:10:05.749800 23319 catalog_manager.cc:5671] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 96695e33efb845dd97f83b3c703f8ee6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96695e33efb845dd97f83b3c703f8ee6 (127.2.197.195). New cstate: current_term: 1 leader_uuid: "96695e33efb845dd97f83b3c703f8ee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4e3309ca61a466a93b6563e4e9ed705" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 39357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96695e33efb845dd97f83b3c703f8ee6" member_type: VOTER last_known_addr { host: "127.2.197.195" port: 41271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42f5c7503a8f4ee8be4b4b58752aab9e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38083 } health_report { overall_health: UNKNOWN } } }
I20260429 14:10:05.760872 23779 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:05.758618 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:56644 (local address 127.2.197.195:41271)
0429 14:10:05.758692 (+ 74us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:05.758698 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:05.758706 (+ 8us) server_negotiation.cc:408] Connection header received
0429 14:10:05.758745 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:05.758746 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:05.758761 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:05.758821 (+ 60us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:10:05.759067 (+ 246us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.759573 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:05.760231 (+ 658us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:05.760369 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:05.760410 (+ 41us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:10:05.760608 (+ 198us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:10:05.760685 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:05.760780 (+ 95us) server_negotiation.cc:300] Negotiation successful
0429 14:10:05.760805 (+ 25us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260429 14:10:05.762709 23461 raft_consensus.cc:1275] T 6d36de0ebbf64e77b71ac9605f83d2c4 P d4e3309ca61a466a93b6563e4e9ed705 [term 1 FOLLOWER]: Refusing update from remote peer 96695e33efb845dd97f83b3c703f8ee6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:10:05.762836 23592 raft_consensus.cc:1275] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 42f5c7503a8f4ee8be4b4b58752aab9e [term 1 FOLLOWER]: Refusing update from remote peer 96695e33efb845dd97f83b3c703f8ee6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:10:05.763054 23793 consensus_queue.cc:1048] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 96695e33efb845dd97f83b3c703f8ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4e3309ca61a466a93b6563e4e9ed705" member_type: VOTER last_known_addr { host: "127.2.197.193" port: 39357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:10:05.763195 23791 consensus_queue.cc:1048] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 96695e33efb845dd97f83b3c703f8ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42f5c7503a8f4ee8be4b4b58752aab9e" member_type: VOTER last_known_addr { host: "127.2.197.194" port: 38083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:10:05.766543 23796 mvcc.cc:204] Tried to move back new op lower bound from 7280524516401205248 to 7280524516348043264. Current Snapshot: MvccSnapshot[applied={T|T < 7280524516401205248}]
I20260429 14:10:05.766634 23797 mvcc.cc:204] Tried to move back new op lower bound from 7280524516401205248 to 7280524516348043264. Current Snapshot: MvccSnapshot[applied={T|T < 7280524516401205248}]
I20260429 14:10:05.770442 23319 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:10:05.770571 23319 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 14:10:05.771391 23319 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 4acc652221a54445b515de7a245526d2: Sending DeleteTablet for 3 replicas of tablet 6d36de0ebbf64e77b71ac9605f83d2c4
I20260429 14:10:05.771898 23441 tablet_service.cc:1558] Processing DeleteTablet for tablet 6d36de0ebbf64e77b71ac9605f83d2c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:10:05 UTC) from {username='slave'} at 127.0.0.1:36430
I20260429 14:10:05.771907 23703 tablet_service.cc:1558] Processing DeleteTablet for tablet 6d36de0ebbf64e77b71ac9605f83d2c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:10:05 UTC) from {username='slave'} at 127.0.0.1:56632
I20260429 14:10:05.771925 23572 tablet_service.cc:1558] Processing DeleteTablet for tablet 6d36de0ebbf64e77b71ac9605f83d2c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 14:10:05 UTC) from {username='slave'} at 127.0.0.1:48072
I20260429 14:10:05.772233 23807 tablet_replica.cc:333] T 6d36de0ebbf64e77b71ac9605f83d2c4 P d4e3309ca61a466a93b6563e4e9ed705: stopping tablet replica
I20260429 14:10:05.772233 23806 tablet_replica.cc:333] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 42f5c7503a8f4ee8be4b4b58752aab9e: stopping tablet replica
I20260429 14:10:05.772331 23807 raft_consensus.cc:2243] T 6d36de0ebbf64e77b71ac9605f83d2c4 P d4e3309ca61a466a93b6563e4e9ed705 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:10:05.772347 23806 raft_consensus.cc:2243] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 42f5c7503a8f4ee8be4b4b58752aab9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 14:10:05.772441 23807 raft_consensus.cc:2272] T 6d36de0ebbf64e77b71ac9605f83d2c4 P d4e3309ca61a466a93b6563e4e9ed705 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:10:05.772460 23806 raft_consensus.cc:2272] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 42f5c7503a8f4ee8be4b4b58752aab9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:10:05.772599 23808 tablet_replica.cc:333] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 96695e33efb845dd97f83b3c703f8ee6: stopping tablet replica
I20260429 14:10:05.772707 23808 raft_consensus.cc:2243] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 96695e33efb845dd97f83b3c703f8ee6 [term 1 LEADER]: Raft consensus shutting down.
I20260429 14:10:05.772838 23808 raft_consensus.cc:2272] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 96695e33efb845dd97f83b3c703f8ee6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 14:10:05.772948 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 23379
I20260429 14:10:05.774191 23808 ts_tablet_manager.cc:1916] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 96695e33efb845dd97f83b3c703f8ee6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:10:05.776242 23808 ts_tablet_manager.cc:1929] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 96695e33efb845dd97f83b3c703f8ee6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 14:10:05.776459 23808 log.cc:1199] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 96695e33efb845dd97f83b3c703f8ee6: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-2/wal/wals/6d36de0ebbf64e77b71ac9605f83d2c4
I20260429 14:10:05.776777 23808 ts_tablet_manager.cc:1950] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 96695e33efb845dd97f83b3c703f8ee6: Deleting consensus metadata
I20260429 14:10:05.777196 23306 catalog_manager.cc:5002] TS 96695e33efb845dd97f83b3c703f8ee6 (127.2.197.195:41271): tablet 6d36de0ebbf64e77b71ac9605f83d2c4 (table test-table [id=3be43436782c4da7b598623cd4e55683]) successfully deleted
I20260429 14:10:05.778630 23806 ts_tablet_manager.cc:1916] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 42f5c7503a8f4ee8be4b4b58752aab9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 14:10:05.780028 23806 ts_tablet_manager.cc:1929] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 42f5c7503a8f4ee8be4b4b58752aab9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 14:10:05.780120 23806 log.cc:1199] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 42f5c7503a8f4ee8be4b4b58752aab9e: Deleting WAL directory at /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777471413505459-2839-0/minicluster-data/ts-1/wal/wals/6d36de0ebbf64e77b71ac9605f83d2c4
I20260429 14:10:05.780385 23806 ts_tablet_manager.cc:1950] T 6d36de0ebbf64e77b71ac9605f83d2c4 P 42f5c7503a8f4ee8be4b4b58752aab9e: Deleting consensus metadata
I20260429 14:10:05.780848 23306 catalog_manager.cc:5002] TS 42f5c7503a8f4ee8be4b4b58752aab9e (127.2.197.194:38083): tablet 6d36de0ebbf64e77b71ac9605f83d2c4 (table test-table [id=3be43436782c4da7b598623cd4e55683]) successfully deleted
W20260429 14:10:05.782394 23304 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.197.193:39357 (error 108)
I20260429 14:10:05.782558 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 23510
I20260429 14:10:05.783582 23777 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:05.783444 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.193:39357 (local address 127.0.0.1:36440)
0429 14:10:05.783513 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:05.783556 (+ 43us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.193:39357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260429 14:10:05.783766 23304 catalog_manager.cc:4729] TS d4e3309ca61a466a93b6563e4e9ed705 (127.2.197.193:39357): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6d36de0ebbf64e77b71ac9605f83d2c4: Network error: Client connection negotiation failed: client connection to 127.2.197.193:39357: connect: Connection refused (error 111)
I20260429 14:10:05.788760 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 23641
I20260429 14:10:05.794466 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 23288
2026-04-29T14:10:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (731 ms)
[----------] 6 tests from EncryptionPolicyTest (11829 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:10:05 dist-test-slave-8n77 krb5kdc[23812](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:10:05 dist-test-slave-8n77 krb5kdc[23812](info): set up 2 sockets
Apr 29 14:10:05 dist-test-slave-8n77 krb5kdc[23812](info): commencing operation
krb5kdc: starting...
W20260429 14:10:07.819739 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.005s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:10:07 dist-test-slave-8n77 krb5kdc[23812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:10:07Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:10:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:10:07.973994 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37955
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:39361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:10:08.053453 23828 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:08.053750 23828 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:08.053791 23828 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:10:08.055481 23828 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:10:08.055599 23828 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:08.055624 23828 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:08.055645 23828 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:10:08.055673 23828 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:10:08.057713 23828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37955
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:08.058049 23828 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:08.058259 23828 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:10:08.061264 23834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:08.061264 23836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:08.061336 23833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:08.061446 23828 server_base.cc:1061] running on GCE node
I20260429 14:10:08.061956 23828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:08.062232 23828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:08.063489 23828 hybrid_clock.cc:648] HybridClock initialized: now 1777471808063482 us; error 41 us; skew 500 ppm
Apr 29 14:10:08 dist-test-slave-8n77 krb5kdc[23812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:10:08.066243 23828 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260429 14:10:08.066910 23828 webserver.cc:492] Webserver started at http://127.0.0.1:33529/ using document root <none> and password file <none>
I20260429 14:10:08.067134 23828 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:08.067178 23828 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:08.067286 23828 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:08.068253 23828 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "b0ed570d76b740daa5ec506199e01c5e"
format_stamp: "Formatted at 2026-04-29 14:10:08 on dist-test-slave-8n77"
server_key: "30f12730c42c69f28f302780b8a1866d"
server_key_iv: "d1c9e1709041704fc60cf8a27ffb039d"
server_key_version: "encryptionkey@0"
I20260429 14:10:08.068581 23828 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "b0ed570d76b740daa5ec506199e01c5e"
format_stamp: "Formatted at 2026-04-29 14:10:08 on dist-test-slave-8n77"
server_key: "30f12730c42c69f28f302780b8a1866d"
server_key_iv: "d1c9e1709041704fc60cf8a27ffb039d"
server_key_version: "encryptionkey@0"
I20260429 14:10:08.069875 23828 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260429 14:10:08.070767 23843 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:08.071010 23828 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260429 14:10:08.071096 23828 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "b0ed570d76b740daa5ec506199e01c5e"
format_stamp: "Formatted at 2026-04-29 14:10:08 on dist-test-slave-8n77"
server_key: "30f12730c42c69f28f302780b8a1866d"
server_key_iv: "d1c9e1709041704fc60cf8a27ffb039d"
server_key_version: "encryptionkey@0"
I20260429 14:10:08.071179 23828 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:08.085732 23828 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:08.088006 23828 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:08.088183 23828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:08.092481 23828 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37955
I20260429 14:10:08.092536 23895 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:37955 every 8 connection(s)
I20260429 14:10:08.092872 23828 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:10:08.093511 23896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:08.095970 23896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0ed570d76b740daa5ec506199e01c5e: Bootstrap starting.
I20260429 14:10:08.096529 23896 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0ed570d76b740daa5ec506199e01c5e: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:08.096731 23896 log.cc:826] T 00000000000000000000000000000000 P b0ed570d76b740daa5ec506199e01c5e: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:08.097455 23896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0ed570d76b740daa5ec506199e01c5e: No bootstrap required, opened a new log
I20260429 14:10:08.098731 23896 raft_consensus.cc:359] T 00000000000000000000000000000000 P b0ed570d76b740daa5ec506199e01c5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ed570d76b740daa5ec506199e01c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37955 } }
I20260429 14:10:08.098850 23896 raft_consensus.cc:385] T 00000000000000000000000000000000 P b0ed570d76b740daa5ec506199e01c5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:10:08.098869 23896 raft_consensus.cc:740] T 00000000000000000000000000000000 P b0ed570d76b740daa5ec506199e01c5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0ed570d76b740daa5ec506199e01c5e, State: Initialized, Role: FOLLOWER
I20260429 14:10:08.099017 23896 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0ed570d76b740daa5ec506199e01c5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ed570d76b740daa5ec506199e01c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37955 } }
I20260429 14:10:08.099092 23896 raft_consensus.cc:399] T 00000000000000000000000000000000 P b0ed570d76b740daa5ec506199e01c5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:10:08.099114 23896 raft_consensus.cc:493] T 00000000000000000000000000000000 P b0ed570d76b740daa5ec506199e01c5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:10:08.099174 23896 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b0ed570d76b740daa5ec506199e01c5e [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:10:08.099244 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 23828
I20260429 14:10:08.099366 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:10:08.099555 2839 external_mini_cluster.cc:1468] Setting key 1adb0d1aee0643d8a51a0daa928bac47
I20260429 14:10:08.099859 23896 raft_consensus.cc:515] T 00000000000000000000000000000000 P b0ed570d76b740daa5ec506199e01c5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ed570d76b740daa5ec506199e01c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37955 } }
I20260429 14:10:08.100194 23896 leader_election.cc:304] T 00000000000000000000000000000000 P b0ed570d76b740daa5ec506199e01c5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0ed570d76b740daa5ec506199e01c5e; no voters:
I20260429 14:10:08.100391 23896 leader_election.cc:290] T 00000000000000000000000000000000 P b0ed570d76b740daa5ec506199e01c5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:10:08.100430 23901 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b0ed570d76b740daa5ec506199e01c5e [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:10:08.100545 23901 raft_consensus.cc:697] T 00000000000000000000000000000000 P b0ed570d76b740daa5ec506199e01c5e [term 1 LEADER]: Becoming Leader. State: Replica: b0ed570d76b740daa5ec506199e01c5e, State: Running, Role: LEADER
I20260429 14:10:08.100620 23901 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0ed570d76b740daa5ec506199e01c5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ed570d76b740daa5ec506199e01c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37955 } }
I20260429 14:10:08.100648 23896 sys_catalog.cc:565] T 00000000000000000000000000000000 P b0ed570d76b740daa5ec506199e01c5e [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:10:08.100910 23901 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0ed570d76b740daa5ec506199e01c5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0ed570d76b740daa5ec506199e01c5e. Latest consensus state: current_term: 1 leader_uuid: "b0ed570d76b740daa5ec506199e01c5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ed570d76b740daa5ec506199e01c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37955 } } }
I20260429 14:10:08.100924 23903 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0ed570d76b740daa5ec506199e01c5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0ed570d76b740daa5ec506199e01c5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ed570d76b740daa5ec506199e01c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37955 } } }
I20260429 14:10:08.100996 23901 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0ed570d76b740daa5ec506199e01c5e [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:08.101054 23903 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0ed570d76b740daa5ec506199e01c5e [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:08.101331 23905 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:10:08.101828 23905 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:10:08.103789 23905 catalog_manager.cc:1357] Generated new cluster ID: 3b00618bbbd94267924c4109be69e4b4
I20260429 14:10:08.103847 23905 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Apr 29 14:10:08 dist-test-slave-8n77 krb5kdc[23812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260429 14:10:08.111680 23900 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:08.100059 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33793 (local address 127.0.0.1:37955)
0429 14:10:08.100209 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:08.100218 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:08.100245 (+ 27us) server_negotiation.cc:408] Connection header received
0429 14:10:08.100398 (+ 153us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:08.100404 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:08.100463 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:08.101870 (+ 1407us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:08.102044 (+ 174us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:08.103823 (+ 1779us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:08.104594 (+ 771us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:08.105274 (+ 680us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:08.107392 (+ 2118us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:08.107422 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:08.107423 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:08.107440 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:08.109302 (+ 1862us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:08.109732 (+ 430us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:08.109735 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:08.109738 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:08.109793 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:08.110025 (+ 232us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:08.110026 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:08.110026 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:08.110236 (+ 210us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:08.110496 (+ 260us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:08.111017 (+ 521us) server_negotiation.cc:300] Negotiation successful
0429 14:10:08.111455 (+ 438us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":84,"threads_started":1}
I20260429 14:10:08.112756 2839 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260429 14:10:08.118702 23905 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:10:08.119421 23905 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:10:08.120735 23900 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:08.114194 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:47953 (local address 127.0.0.1:37955)
0429 14:10:08.114238 (+ 44us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:08.114240 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:08.114407 (+ 167us) server_negotiation.cc:408] Connection header received
0429 14:10:08.114483 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:08.114484 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:08.114494 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:08.114548 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:08.115258 (+ 710us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:08.115741 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:08.116311 (+ 570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:08.116524 (+ 213us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:08.117546 (+ 1022us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:08.117549 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:08.117550 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:08.117556 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:08.119158 (+ 1602us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:08.119617 (+ 459us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:08.119618 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:08.119618 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:08.119647 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:08.120140 (+ 493us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:08.120142 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:08.120143 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:08.120247 (+ 104us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:08.120404 (+ 157us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:08.120564 (+ 160us) server_negotiation.cc:300] Negotiation successful
0429 14:10:08.120632 (+ 68us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260429 14:10:08.126559 23905 catalog_manager.cc:6044] T 00000000000000000000000000000000 P b0ed570d76b740daa5ec506199e01c5e: Generated new TSK 0
I20260429 14:10:08.126755 23905 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 14:10:08.131536 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 23828
W20260429 14:10:08.135807 23810 connection.cc:570] client connection to 127.0.0.1:37955 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37955): Connection reset by peer (error 104)
2026-04-29T14:10:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2334 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:10:08 dist-test-slave-8n77 krb5kdc[23927](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:10:08 dist-test-slave-8n77 krb5kdc[23927](info): set up 2 sockets
Apr 29 14:10:08 dist-test-slave-8n77 krb5kdc[23927](info): commencing operation
krb5kdc: starting...
W20260429 14:10:10.175875 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:10:10 dist-test-slave-8n77 krb5kdc[23927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:10:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:10:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:10:10.327800 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36981
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:42275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:10:10.408497 23943 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:10.408733 23943 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:10.408759 23943 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:10:10.410482 23943 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:10:10.410539 23943 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:10.410554 23943 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:10.410567 23943 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:10:10.410579 23943 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:10:10.412360 23943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36981
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:10.412633 23943 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:10.412824 23943 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:10:10.415584 23949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:10.415589 23951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:10.415619 23948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:10.416162 23943 server_base.cc:1061] running on GCE node
I20260429 14:10:10.416483 23943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:10.416743 23943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:10.417914 23943 hybrid_clock.cc:648] HybridClock initialized: now 1777471810417903 us; error 48 us; skew 500 ppm
Apr 29 14:10:10 dist-test-slave-8n77 krb5kdc[23927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:10:10.420598 23943 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260429 14:10:10.421130 23943 webserver.cc:492] Webserver started at http://127.0.0.1:41279/ using document root <none> and password file <none>
I20260429 14:10:10.421370 23943 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:10.421437 23943 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:10.421555 23943 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:10.422587 23943 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "00feadca536d48ecb2de0819e1906042"
format_stamp: "Formatted at 2026-04-29 14:10:10 on dist-test-slave-8n77"
server_key: "ce17ee774bf370dd191a55c69366d76a"
server_key_iv: "5d04b6c5a0c3a8c0f4346aa48b8a01eb"
server_key_version: "encryptionkey@0"
I20260429 14:10:10.422930 23943 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "00feadca536d48ecb2de0819e1906042"
format_stamp: "Formatted at 2026-04-29 14:10:10 on dist-test-slave-8n77"
server_key: "ce17ee774bf370dd191a55c69366d76a"
server_key_iv: "5d04b6c5a0c3a8c0f4346aa48b8a01eb"
server_key_version: "encryptionkey@0"
I20260429 14:10:10.424186 23943 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:10:10.425050 23958 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:10.425274 23943 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:10:10.425361 23943 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "00feadca536d48ecb2de0819e1906042"
format_stamp: "Formatted at 2026-04-29 14:10:10 on dist-test-slave-8n77"
server_key: "ce17ee774bf370dd191a55c69366d76a"
server_key_iv: "5d04b6c5a0c3a8c0f4346aa48b8a01eb"
server_key_version: "encryptionkey@0"
I20260429 14:10:10.425444 23943 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:10.438509 23943 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:10.444226 23943 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:10.444363 23943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:10.449074 23943 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36981
I20260429 14:10:10.449162 24010 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:36981 every 8 connection(s)
I20260429 14:10:10.449481 23943 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:10:10.450112 24011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:10.452536 24011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00feadca536d48ecb2de0819e1906042: Bootstrap starting.
I20260429 14:10:10.452982 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 23943
I20260429 14:10:10.453074 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:10:10.453274 24011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00feadca536d48ecb2de0819e1906042: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:10.453372 2839 external_mini_cluster.cc:1468] Setting key e43dc45d61d95af733307fecb94cfd40
I20260429 14:10:10.453606 24011 log.cc:826] T 00000000000000000000000000000000 P 00feadca536d48ecb2de0819e1906042: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:10.454490 24011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00feadca536d48ecb2de0819e1906042: No bootstrap required, opened a new log
I20260429 14:10:10.455842 24011 raft_consensus.cc:359] T 00000000000000000000000000000000 P 00feadca536d48ecb2de0819e1906042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00feadca536d48ecb2de0819e1906042" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36981 } }
I20260429 14:10:10.455981 24011 raft_consensus.cc:385] T 00000000000000000000000000000000 P 00feadca536d48ecb2de0819e1906042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:10:10.456025 24011 raft_consensus.cc:740] T 00000000000000000000000000000000 P 00feadca536d48ecb2de0819e1906042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00feadca536d48ecb2de0819e1906042, State: Initialized, Role: FOLLOWER
I20260429 14:10:10.456126 24011 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00feadca536d48ecb2de0819e1906042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00feadca536d48ecb2de0819e1906042" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36981 } }
I20260429 14:10:10.456223 24011 raft_consensus.cc:399] T 00000000000000000000000000000000 P 00feadca536d48ecb2de0819e1906042 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:10:10.456266 24011 raft_consensus.cc:493] T 00000000000000000000000000000000 P 00feadca536d48ecb2de0819e1906042 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:10:10.456336 24011 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 00feadca536d48ecb2de0819e1906042 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:10:10.456903 24011 raft_consensus.cc:515] T 00000000000000000000000000000000 P 00feadca536d48ecb2de0819e1906042 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00feadca536d48ecb2de0819e1906042" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36981 } }
I20260429 14:10:10.457063 24011 leader_election.cc:304] T 00000000000000000000000000000000 P 00feadca536d48ecb2de0819e1906042 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00feadca536d48ecb2de0819e1906042; no voters:
I20260429 14:10:10.457233 24011 leader_election.cc:290] T 00000000000000000000000000000000 P 00feadca536d48ecb2de0819e1906042 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:10:10.457305 24016 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 00feadca536d48ecb2de0819e1906042 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:10:10.457480 24016 raft_consensus.cc:697] T 00000000000000000000000000000000 P 00feadca536d48ecb2de0819e1906042 [term 1 LEADER]: Becoming Leader. State: Replica: 00feadca536d48ecb2de0819e1906042, State: Running, Role: LEADER
I20260429 14:10:10.457510 24011 sys_catalog.cc:565] T 00000000000000000000000000000000 P 00feadca536d48ecb2de0819e1906042 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:10:10.457577 24016 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00feadca536d48ecb2de0819e1906042 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00feadca536d48ecb2de0819e1906042" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36981 } }
I20260429 14:10:10.457878 24016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00feadca536d48ecb2de0819e1906042 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00feadca536d48ecb2de0819e1906042. Latest consensus state: current_term: 1 leader_uuid: "00feadca536d48ecb2de0819e1906042" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00feadca536d48ecb2de0819e1906042" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36981 } } }
I20260429 14:10:10.457996 24016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00feadca536d48ecb2de0819e1906042 [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:10.457885 24017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00feadca536d48ecb2de0819e1906042 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00feadca536d48ecb2de0819e1906042" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00feadca536d48ecb2de0819e1906042" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36981 } } }
I20260429 14:10:10.458159 24017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00feadca536d48ecb2de0819e1906042 [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:10.458469 24015 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:10.453930 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48901 (local address 127.0.0.1:36981)
0429 14:10:10.454129 (+ 199us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:10.454142 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:10.454169 (+ 27us) server_negotiation.cc:408] Connection header received
0429 14:10:10.454568 (+ 399us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:10.454595 (+ 27us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:10.454653 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:10.454755 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:10.455232 (+ 477us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:10.455951 (+ 719us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:10.456739 (+ 788us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:10.457030 (+ 291us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:10.457610 (+ 580us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:10.457619 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:10.457619 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:10:10.457833 (+ 214us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:10.457855 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:10:10.457864 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:10.458209 (+ 345us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:10.458267 (+ 58us) server_negotiation.cc:300] Negotiation successful
0429 14:10:10.458318 (+ 51us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":104,"threads_started":1}
I20260429 14:10:10.459167 24022 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:10:10.459425 2839 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260429 14:10:10.459846 24022 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:10:10.461952 24022 catalog_manager.cc:1357] Generated new cluster ID: d21fc4e3e1a54ee6af31a0158b423726
I20260429 14:10:10.462042 24022 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:10:10.464826 24015 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:10.461664 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:46333 (local address 127.0.0.1:36981)
0429 14:10:10.461708 (+ 44us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:10.461711 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:10.461721 (+ 10us) server_negotiation.cc:408] Connection header received
0429 14:10:10.461820 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:10.461821 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:10.461834 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:10.461902 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:10.462368 (+ 466us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:10.463077 (+ 709us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:10.463868 (+ 791us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:10.464066 (+ 198us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:10.464545 (+ 479us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:10.464549 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:10.464549 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:10:10.464554 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:10.464570 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:10:10.464574 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:10.464638 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:10.464738 (+ 100us) server_negotiation.cc:300] Negotiation successful
0429 14:10:10.464750 (+ 12us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260429 14:10:10.473707 24022 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:10:10.474225 24022 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:10:10.478920 24022 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 00feadca536d48ecb2de0819e1906042: Generated new TSK 0
I20260429 14:10:10.479116 24022 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 14:10:10.481971 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 23943
W20260429 14:10:10.486215 23925 connection.cc:570] client connection to 127.0.0.1:36981 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36981): Connection reset by peer (error 104)
2026-04-29T14:10:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2350 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:10:10 dist-test-slave-8n77 krb5kdc[24042](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:10:10 dist-test-slave-8n77 krb5kdc[24042](info): set up 2 sockets
Apr 29 14:10:10 dist-test-slave-8n77 krb5kdc[24042](info): commencing operation
krb5kdc: starting...
W20260429 14:10:12.511713 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:10:12 dist-test-slave-8n77 krb5kdc[24042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:10:12Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:10:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:10:12.658084 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40067
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:42569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:10:12.739174 24058 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:12.739471 24058 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:12.739498 24058 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:10:12.740988 24058 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:10:12.741036 24058 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:12.741051 24058 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:12.741065 24058 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:10:12.741076 24058 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:10:12.742745 24058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40067
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:12.742954 24058 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:12.743094 24058 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:10:12.745950 24064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:12.745981 24063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:12.745949 24066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:12.746416 24058 server_base.cc:1061] running on GCE node
I20260429 14:10:12.746644 24058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:12.746853 24058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:12.748031 24058 hybrid_clock.cc:648] HybridClock initialized: now 1777471812748018 us; error 32 us; skew 500 ppm
Apr 29 14:10:12 dist-test-slave-8n77 krb5kdc[24042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:10:12.750696 24058 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260429 14:10:12.751317 24058 webserver.cc:492] Webserver started at http://127.0.0.1:34339/ using document root <none> and password file <none>
I20260429 14:10:12.751531 24058 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:12.751597 24058 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:12.751709 24058 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:12.752722 24058 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "4f397c25b46e42fd9d3aac511476d73b"
format_stamp: "Formatted at 2026-04-29 14:10:12 on dist-test-slave-8n77"
server_key: "61b7cd1bfd7347a194c468bc63937027"
server_key_iv: "8b6a7db7985802a82aa55818da47d095"
server_key_version: "encryptionkey@0"
I20260429 14:10:12.753069 24058 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "4f397c25b46e42fd9d3aac511476d73b"
format_stamp: "Formatted at 2026-04-29 14:10:12 on dist-test-slave-8n77"
server_key: "61b7cd1bfd7347a194c468bc63937027"
server_key_iv: "8b6a7db7985802a82aa55818da47d095"
server_key_version: "encryptionkey@0"
I20260429 14:10:12.754549 24058 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260429 14:10:12.755295 24073 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:12.755517 24058 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:10:12.755599 24058 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "4f397c25b46e42fd9d3aac511476d73b"
format_stamp: "Formatted at 2026-04-29 14:10:12 on dist-test-slave-8n77"
server_key: "61b7cd1bfd7347a194c468bc63937027"
server_key_iv: "8b6a7db7985802a82aa55818da47d095"
server_key_version: "encryptionkey@0"
I20260429 14:10:12.755684 24058 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:12.759006 24058 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:12.761209 24058 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:12.761407 24058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:12.767810 24058 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40067
I20260429 14:10:12.767918 24125 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:40067 every 8 connection(s)
I20260429 14:10:12.768548 24058 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:10:12.769213 24126 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:12.771875 24126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f397c25b46e42fd9d3aac511476d73b: Bootstrap starting.
I20260429 14:10:12.772447 24126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f397c25b46e42fd9d3aac511476d73b: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:12.772720 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 24058
I20260429 14:10:12.772751 24126 log.cc:826] T 00000000000000000000000000000000 P 4f397c25b46e42fd9d3aac511476d73b: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:12.772846 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:10:12.773049 2839 external_mini_cluster.cc:1468] Setting key 4b9de731d7596d8bbeee429649b95a0d
I20260429 14:10:12.773576 24126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f397c25b46e42fd9d3aac511476d73b: No bootstrap required, opened a new log
I20260429 14:10:12.774974 24126 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4f397c25b46e42fd9d3aac511476d73b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f397c25b46e42fd9d3aac511476d73b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40067 } }
I20260429 14:10:12.775101 24126 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4f397c25b46e42fd9d3aac511476d73b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:10:12.775122 24126 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4f397c25b46e42fd9d3aac511476d73b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f397c25b46e42fd9d3aac511476d73b, State: Initialized, Role: FOLLOWER
I20260429 14:10:12.775236 24126 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f397c25b46e42fd9d3aac511476d73b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f397c25b46e42fd9d3aac511476d73b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40067 } }
I20260429 14:10:12.775318 24126 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4f397c25b46e42fd9d3aac511476d73b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:10:12.775359 24126 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4f397c25b46e42fd9d3aac511476d73b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:10:12.775398 24126 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4f397c25b46e42fd9d3aac511476d73b [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:10:12.776193 24126 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4f397c25b46e42fd9d3aac511476d73b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f397c25b46e42fd9d3aac511476d73b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40067 } }
I20260429 14:10:12.776348 24126 leader_election.cc:304] T 00000000000000000000000000000000 P 4f397c25b46e42fd9d3aac511476d73b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f397c25b46e42fd9d3aac511476d73b; no voters:
I20260429 14:10:12.776515 24126 leader_election.cc:290] T 00000000000000000000000000000000 P 4f397c25b46e42fd9d3aac511476d73b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:10:12.776582 24131 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4f397c25b46e42fd9d3aac511476d73b [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:10:12.776717 24131 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4f397c25b46e42fd9d3aac511476d73b [term 1 LEADER]: Becoming Leader. State: Replica: 4f397c25b46e42fd9d3aac511476d73b, State: Running, Role: LEADER
I20260429 14:10:12.776727 24130 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:12.773586 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:50707 (local address 127.0.0.1:40067)
0429 14:10:12.773741 (+ 155us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:12.773754 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:12.773792 (+ 38us) server_negotiation.cc:408] Connection header received
0429 14:10:12.775027 (+ 1235us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:12.775043 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:12.775248 (+ 205us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:12.775435 (+ 187us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:12.776132 (+ 697us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:12.776134 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:12.776135 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:10:12.776138 (+ 3us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:12.776163 (+ 25us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:10:12.776169 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:12.776224 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:12.776478 (+ 254us) server_negotiation.cc:300] Negotiation successful
0429 14:10:12.776558 (+ 80us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":79,"threads_started":1}
I20260429 14:10:12.776819 24131 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f397c25b46e42fd9d3aac511476d73b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f397c25b46e42fd9d3aac511476d73b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40067 } }
I20260429 14:10:12.776753 24126 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4f397c25b46e42fd9d3aac511476d73b [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:10:12.777146 24132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f397c25b46e42fd9d3aac511476d73b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f397c25b46e42fd9d3aac511476d73b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f397c25b46e42fd9d3aac511476d73b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40067 } } }
I20260429 14:10:12.777223 24132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f397c25b46e42fd9d3aac511476d73b [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:12.777134 24131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f397c25b46e42fd9d3aac511476d73b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f397c25b46e42fd9d3aac511476d73b. Latest consensus state: current_term: 1 leader_uuid: "4f397c25b46e42fd9d3aac511476d73b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f397c25b46e42fd9d3aac511476d73b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40067 } } }
I20260429 14:10:12.777513 24131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f397c25b46e42fd9d3aac511476d73b [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:12.777813 2839 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260429 14:10:12.778035 24135 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:10:12.778574 24135 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:10:12.780373 24135 catalog_manager.cc:1357] Generated new cluster ID: 5ff72054acd7425eb098c59f9da4a9ec
I20260429 14:10:12.780443 24135 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:10:12.783071 24130 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:12.779440 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53751 (local address 127.0.0.1:40067)
0429 14:10:12.779474 (+ 34us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:12.779475 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:12.779780 (+ 305us) server_negotiation.cc:408] Connection header received
0429 14:10:12.779847 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:12.779848 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:12.779853 (+ 5us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:12.779928 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:12.782788 (+ 2860us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:12.782791 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:12.782791 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:10:12.782797 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:12.782818 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:10:12.782823 (+ 5us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:12.782885 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:12.782990 (+ 105us) server_negotiation.cc:300] Negotiation successful
0429 14:10:12.783006 (+ 16us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260429 14:10:12.789917 24135 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:10:12.790091 24135 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:10:12.799070 24135 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 4f397c25b46e42fd9d3aac511476d73b: Generated new TSK 0
I20260429 14:10:12.799597 24135 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 14:10:12.806123 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 24058
2026-04-29T14:10:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:10:12 dist-test-slave-8n77 krb5kdc[24157](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:10:12 dist-test-slave-8n77 krb5kdc[24157](info): set up 2 sockets
Apr 29 14:10:12 dist-test-slave-8n77 krb5kdc[24157](info): commencing operation
krb5kdc: starting...
W20260429 14:10:14.848363 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.023s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:10:14 dist-test-slave-8n77 krb5kdc[24157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:10:14Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:10:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:10:14.993955 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34769
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:42363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:10:15.072582 24173 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:15.072831 24173 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:15.072870 24173 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:10:15.074457 24173 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:10:15.074544 24173 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:15.074569 24173 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:15.074599 24173 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:10:15.074630 24173 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:10:15.076362 24173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:15.076658 24173 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:15.076864 24173 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:10:15.079362 24181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:15.079349 24178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:15.079367 24179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:15.079751 24173 server_base.cc:1061] running on GCE node
I20260429 14:10:15.079952 24173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:15.080180 24173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:15.081341 24173 hybrid_clock.cc:648] HybridClock initialized: now 1777471815081327 us; error 31 us; skew 500 ppm
Apr 29 14:10:15 dist-test-slave-8n77 krb5kdc[24157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:10:15.083887 24173 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260429 14:10:15.084519 24173 webserver.cc:492] Webserver started at http://127.0.0.1:44877/ using document root <none> and password file <none>
I20260429 14:10:15.084722 24173 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:15.084764 24173 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:15.084837 24173 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:15.085870 24173 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "6c0657e4e7204185809d37d40487a9c9"
format_stamp: "Formatted at 2026-04-29 14:10:15 on dist-test-slave-8n77"
server_key: "7cd7a103f88b994ba3afa179718c436a"
server_key_iv: "879f2da6e11a12deb010827cfc8611c3"
server_key_version: "encryptionkey@0"
I20260429 14:10:15.086205 24173 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "6c0657e4e7204185809d37d40487a9c9"
format_stamp: "Formatted at 2026-04-29 14:10:15 on dist-test-slave-8n77"
server_key: "7cd7a103f88b994ba3afa179718c436a"
server_key_iv: "879f2da6e11a12deb010827cfc8611c3"
server_key_version: "encryptionkey@0"
I20260429 14:10:15.087536 24173 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:10:15.088428 24188 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:15.088632 24173 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:10:15.088704 24173 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "6c0657e4e7204185809d37d40487a9c9"
format_stamp: "Formatted at 2026-04-29 14:10:15 on dist-test-slave-8n77"
server_key: "7cd7a103f88b994ba3afa179718c436a"
server_key_iv: "879f2da6e11a12deb010827cfc8611c3"
server_key_version: "encryptionkey@0"
I20260429 14:10:15.088781 24173 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:15.097307 24173 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:15.099391 24173 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:15.099546 24173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:15.104174 24173 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34769
I20260429 14:10:15.104257 24240 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:34769 every 8 connection(s)
I20260429 14:10:15.104545 24173 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:10:15.105139 24241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:15.107615 24241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c0657e4e7204185809d37d40487a9c9: Bootstrap starting.
I20260429 14:10:15.108139 24241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c0657e4e7204185809d37d40487a9c9: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:15.108341 24241 log.cc:826] T 00000000000000000000000000000000 P 6c0657e4e7204185809d37d40487a9c9: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:15.108752 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 24173
I20260429 14:10:15.108886 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:10:15.109061 2839 external_mini_cluster.cc:1468] Setting key 56fd8b29d2a1b36189858b535ba66940
I20260429 14:10:15.109105 24241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c0657e4e7204185809d37d40487a9c9: No bootstrap required, opened a new log
I20260429 14:10:15.110735 24241 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6c0657e4e7204185809d37d40487a9c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0657e4e7204185809d37d40487a9c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34769 } }
I20260429 14:10:15.110877 24241 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6c0657e4e7204185809d37d40487a9c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:10:15.110937 24241 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6c0657e4e7204185809d37d40487a9c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c0657e4e7204185809d37d40487a9c9, State: Initialized, Role: FOLLOWER
I20260429 14:10:15.111155 24241 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c0657e4e7204185809d37d40487a9c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0657e4e7204185809d37d40487a9c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34769 } }
I20260429 14:10:15.111305 24241 raft_consensus.cc:399] T 00000000000000000000000000000000 P 6c0657e4e7204185809d37d40487a9c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:10:15.111358 24241 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6c0657e4e7204185809d37d40487a9c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:10:15.111413 24241 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6c0657e4e7204185809d37d40487a9c9 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:10:15.111984 24241 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6c0657e4e7204185809d37d40487a9c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0657e4e7204185809d37d40487a9c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34769 } }
I20260429 14:10:15.112121 24241 leader_election.cc:304] T 00000000000000000000000000000000 P 6c0657e4e7204185809d37d40487a9c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c0657e4e7204185809d37d40487a9c9; no voters:
I20260429 14:10:15.112355 24241 leader_election.cc:290] T 00000000000000000000000000000000 P 6c0657e4e7204185809d37d40487a9c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:10:15.112387 24246 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6c0657e4e7204185809d37d40487a9c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:10:15.112548 24246 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6c0657e4e7204185809d37d40487a9c9 [term 1 LEADER]: Becoming Leader. State: Replica: 6c0657e4e7204185809d37d40487a9c9, State: Running, Role: LEADER
I20260429 14:10:15.112646 24241 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6c0657e4e7204185809d37d40487a9c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:10:15.112676 24246 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c0657e4e7204185809d37d40487a9c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0657e4e7204185809d37d40487a9c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34769 } }
I20260429 14:10:15.113062 24246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c0657e4e7204185809d37d40487a9c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c0657e4e7204185809d37d40487a9c9. Latest consensus state: current_term: 1 leader_uuid: "6c0657e4e7204185809d37d40487a9c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0657e4e7204185809d37d40487a9c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34769 } } }
I20260429 14:10:15.113197 24246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c0657e4e7204185809d37d40487a9c9 [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:15.113453 24254 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:10:15.113955 24254 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:10:15.115643 24247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c0657e4e7204185809d37d40487a9c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c0657e4e7204185809d37d40487a9c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0657e4e7204185809d37d40487a9c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34769 } } }
I20260429 14:10:15.115887 24247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c0657e4e7204185809d37d40487a9c9 [sys.catalog]: This master's current role is: LEADER
Apr 29 14:10:15 dist-test-slave-8n77 krb5kdc[24157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260429 14:10:15.116880 24254 catalog_manager.cc:1357] Generated new cluster ID: 17285a4b7fa640f7941a7766c5067f6a
I20260429 14:10:15.116930 24254 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:10:15.120828 24245 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:15.109641 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37537 (local address 127.0.0.1:34769)
0429 14:10:15.109811 (+ 170us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:15.109820 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:15.109837 (+ 17us) server_negotiation.cc:408] Connection header received
0429 14:10:15.110720 (+ 883us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:15.110728 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:15.110784 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:15.110936 (+ 152us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:15.111879 (+ 943us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:15.112965 (+ 1086us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:15.113811 (+ 846us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:15.114323 (+ 512us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:15.116938 (+ 2615us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:15.116944 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:15.116945 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:15.116958 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:15.119049 (+ 2091us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:15.119472 (+ 423us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:15.119475 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:15.119476 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:15.119529 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:15.119692 (+ 163us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:15.119693 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:15.119693 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:15.119931 (+ 238us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:15.120056 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:15.120518 (+ 462us) server_negotiation.cc:300] Negotiation successful
0429 14:10:15.120645 (+ 127us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
I20260429 14:10:15.121647 2839 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260429 14:10:15.130080 24245 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:15.123201 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60837 (local address 127.0.0.1:34769)
0429 14:10:15.123237 (+ 36us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:15.123238 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:15.123348 (+ 110us) server_negotiation.cc:408] Connection header received
0429 14:10:15.123392 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:15.123393 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:15.123404 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:15.123468 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:15.124237 (+ 769us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:15.124988 (+ 751us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:15.125689 (+ 701us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:15.125877 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:15.127310 (+ 1433us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:15.127313 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:15.127314 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:15.127322 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:15.129018 (+ 1696us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:15.129369 (+ 351us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:15.129370 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:15.129370 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:15.129394 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:15.129551 (+ 157us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:15.129553 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:15.129553 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:15.129661 (+ 108us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:15.129759 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:15.129895 (+ 136us) server_negotiation.cc:300] Negotiation successful
0429 14:10:15.129978 (+ 83us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260429 14:10:15.141659 24254 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:10:15.142510 24254 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:10:15.146770 24254 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 6c0657e4e7204185809d37d40487a9c9: Generated new TSK 0
I20260429 14:10:15.146956 24254 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 14:10:15.149323 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 24173
2026-04-29T14:10:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2342 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:10:15 dist-test-slave-8n77 krb5kdc[24272](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:10:15 dist-test-slave-8n77 krb5kdc[24272](info): set up 2 sockets
Apr 29 14:10:15 dist-test-slave-8n77 krb5kdc[24272](info): commencing operation
krb5kdc: starting...
W20260429 14:10:17.167874 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.000s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:10:17 dist-test-slave-8n77 krb5kdc[24272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:10:17Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:10:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:10:17.310400 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41035
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:36471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:10:17.388981 24288 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:17.389214 24288 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:17.389250 24288 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:10:17.390764 24288 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:10:17.390862 24288 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:17.390888 24288 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:17.390918 24288 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:10:17.390951 24288 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:10:17.392599 24288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:17.392872 24288 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:17.393064 24288 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:10:17.395707 24293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:17.395735 24296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:17.395707 24294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:17.396355 24288 server_base.cc:1061] running on GCE node
I20260429 14:10:17.396602 24288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:17.396838 24288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:17.397980 24288 hybrid_clock.cc:648] HybridClock initialized: now 1777471817397966 us; error 31 us; skew 500 ppm
Apr 29 14:10:17 dist-test-slave-8n77 krb5kdc[24272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:10:17.400449 24288 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260429 14:10:17.400911 24288 webserver.cc:492] Webserver started at http://127.0.0.1:40787/ using document root <none> and password file <none>
I20260429 14:10:17.401063 24288 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:17.401098 24288 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:17.401165 24288 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:17.402037 24288 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "b55cff471a5d40698daf52c4614d0972"
format_stamp: "Formatted at 2026-04-29 14:10:17 on dist-test-slave-8n77"
server_key: "2628b3f5f994be33590565db2d8c8430"
server_key_iv: "c7075fca0eb5f0548924a6d534101078"
server_key_version: "encryptionkey@0"
I20260429 14:10:17.402381 24288 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "b55cff471a5d40698daf52c4614d0972"
format_stamp: "Formatted at 2026-04-29 14:10:17 on dist-test-slave-8n77"
server_key: "2628b3f5f994be33590565db2d8c8430"
server_key_iv: "c7075fca0eb5f0548924a6d534101078"
server_key_version: "encryptionkey@0"
I20260429 14:10:17.403569 24288 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:10:17.404248 24303 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:17.404484 24288 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:10:17.404589 24288 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "b55cff471a5d40698daf52c4614d0972"
format_stamp: "Formatted at 2026-04-29 14:10:17 on dist-test-slave-8n77"
server_key: "2628b3f5f994be33590565db2d8c8430"
server_key_iv: "c7075fca0eb5f0548924a6d534101078"
server_key_version: "encryptionkey@0"
I20260429 14:10:17.404670 24288 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:17.417861 24288 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:17.424270 24288 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:17.424451 24288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:17.428668 24288 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41035
I20260429 14:10:17.428711 24355 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:41035 every 8 connection(s)
I20260429 14:10:17.429045 24288 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:10:17.429656 24356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:17.432045 24356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b55cff471a5d40698daf52c4614d0972: Bootstrap starting.
I20260429 14:10:17.432646 24356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b55cff471a5d40698daf52c4614d0972: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:17.432902 24356 log.cc:826] T 00000000000000000000000000000000 P b55cff471a5d40698daf52c4614d0972: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:17.433712 24356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b55cff471a5d40698daf52c4614d0972: No bootstrap required, opened a new log
I20260429 14:10:17.434998 24356 raft_consensus.cc:359] T 00000000000000000000000000000000 P b55cff471a5d40698daf52c4614d0972 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55cff471a5d40698daf52c4614d0972" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41035 } }
I20260429 14:10:17.435117 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 24288
I20260429 14:10:17.435140 24356 raft_consensus.cc:385] T 00000000000000000000000000000000 P b55cff471a5d40698daf52c4614d0972 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:10:17.435182 24356 raft_consensus.cc:740] T 00000000000000000000000000000000 P b55cff471a5d40698daf52c4614d0972 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b55cff471a5d40698daf52c4614d0972, State: Initialized, Role: FOLLOWER
I20260429 14:10:17.435232 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:10:17.435439 2839 external_mini_cluster.cc:1468] Setting key 0c0299dfd3be9419732f4ff107a6ae1a
I20260429 14:10:17.435511 24356 consensus_queue.cc:260] T 00000000000000000000000000000000 P b55cff471a5d40698daf52c4614d0972 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55cff471a5d40698daf52c4614d0972" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41035 } }
I20260429 14:10:17.435636 24356 raft_consensus.cc:399] T 00000000000000000000000000000000 P b55cff471a5d40698daf52c4614d0972 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:10:17.435678 24356 raft_consensus.cc:493] T 00000000000000000000000000000000 P b55cff471a5d40698daf52c4614d0972 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:10:17.435727 24356 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b55cff471a5d40698daf52c4614d0972 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:10:17.436510 24356 raft_consensus.cc:515] T 00000000000000000000000000000000 P b55cff471a5d40698daf52c4614d0972 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55cff471a5d40698daf52c4614d0972" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41035 } }
I20260429 14:10:17.436630 24356 leader_election.cc:304] T 00000000000000000000000000000000 P b55cff471a5d40698daf52c4614d0972 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b55cff471a5d40698daf52c4614d0972; no voters:
I20260429 14:10:17.436774 24356 leader_election.cc:290] T 00000000000000000000000000000000 P b55cff471a5d40698daf52c4614d0972 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:10:17.436837 24361 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b55cff471a5d40698daf52c4614d0972 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:10:17.436996 24356 sys_catalog.cc:565] T 00000000000000000000000000000000 P b55cff471a5d40698daf52c4614d0972 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:10:17.437001 24361 raft_consensus.cc:697] T 00000000000000000000000000000000 P b55cff471a5d40698daf52c4614d0972 [term 1 LEADER]: Becoming Leader. State: Replica: b55cff471a5d40698daf52c4614d0972, State: Running, Role: LEADER
I20260429 14:10:17.437170 24361 consensus_queue.cc:237] T 00000000000000000000000000000000 P b55cff471a5d40698daf52c4614d0972 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55cff471a5d40698daf52c4614d0972" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41035 } }
I20260429 14:10:17.437588 24363 sys_catalog.cc:455] T 00000000000000000000000000000000 P b55cff471a5d40698daf52c4614d0972 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b55cff471a5d40698daf52c4614d0972. Latest consensus state: current_term: 1 leader_uuid: "b55cff471a5d40698daf52c4614d0972" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55cff471a5d40698daf52c4614d0972" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41035 } } }
I20260429 14:10:17.437662 24363 sys_catalog.cc:458] T 00000000000000000000000000000000 P b55cff471a5d40698daf52c4614d0972 [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:17.437856 24362 sys_catalog.cc:455] T 00000000000000000000000000000000 P b55cff471a5d40698daf52c4614d0972 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b55cff471a5d40698daf52c4614d0972" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55cff471a5d40698daf52c4614d0972" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41035 } } }
I20260429 14:10:17.438148 24362 sys_catalog.cc:458] T 00000000000000000000000000000000 P b55cff471a5d40698daf52c4614d0972 [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:17.438301 24365 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:10:17.438840 24365 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:10:17.440554 24360 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:17.435971 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:58739 (local address 127.0.0.1:41035)
0429 14:10:17.436146 (+ 175us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:17.436158 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:17.436185 (+ 27us) server_negotiation.cc:408] Connection header received
0429 14:10:17.436296 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:17.436302 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:17.436350 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:17.436483 (+ 133us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:17.436977 (+ 494us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:17.438262 (+ 1285us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:17.439419 (+ 1157us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:17.439662 (+ 243us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:17.439916 (+ 254us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:17.439923 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:17.439923 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:10:17.440175 (+ 252us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:17.440194 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:10:17.440203 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:17.440298 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:17.440375 (+ 77us) server_negotiation.cc:300] Negotiation successful
0429 14:10:17.440412 (+ 37us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
I20260429 14:10:17.440822 24365 catalog_manager.cc:1357] Generated new cluster ID: 4d8e392fbe264315a6ab9495a166ac08
I20260429 14:10:17.440866 24365 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:10:17.441462 2839 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260429 14:10:17.445772 24360 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:17.443030 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57887 (local address 127.0.0.1:41035)
0429 14:10:17.443068 (+ 38us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:17.443070 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:17.443077 (+ 7us) server_negotiation.cc:408] Connection header received
0429 14:10:17.443115 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:17.443116 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:17.443127 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:17.443183 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:17.443500 (+ 317us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:17.444239 (+ 739us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:17.444929 (+ 690us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:17.445467 (+ 538us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:17.445513 (+ 46us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:17.445515 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:17.445516 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:10:17.445520 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:17.445531 (+ 11us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:10:17.445534 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:17.445594 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:17.445668 (+ 74us) server_negotiation.cc:300] Negotiation successful
0429 14:10:17.445694 (+ 26us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260429 14:10:17.456887 24365 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:10:17.457510 24365 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:10:17.465533 24365 catalog_manager.cc:6044] T 00000000000000000000000000000000 P b55cff471a5d40698daf52c4614d0972: Generated new TSK 0
I20260429 14:10:17.465727 24365 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 14:10:17.470041 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 24288
2026-04-29T14:10:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2320 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:10:17 dist-test-slave-8n77 krb5kdc[24387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:10:17 dist-test-slave-8n77 krb5kdc[24387](info): set up 2 sockets
Apr 29 14:10:17 dist-test-slave-8n77 krb5kdc[24387](info): commencing operation
krb5kdc: starting...
W20260429 14:10:19.495770 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.008s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:10:19 dist-test-slave-8n77 krb5kdc[24387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:10:19Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:10:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:10:19.646775 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45289
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:34445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:10:19.724994 24403 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:19.725194 24403 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:19.725219 24403 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:10:19.726778 24403 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:10:19.726826 24403 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:19.726841 24403 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:19.726854 24403 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:10:19.726866 24403 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:10:19.728579 24403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:19.728787 24403 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:19.728924 24403 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:10:19.731633 24408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:19.731652 24409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:19.731791 24403 server_base.cc:1061] running on GCE node
W20260429 14:10:19.731633 24411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:19.732412 24403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:19.732713 24403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:19.733881 24403 hybrid_clock.cc:648] HybridClock initialized: now 1777471819733855 us; error 45 us; skew 500 ppm
Apr 29 14:10:19 dist-test-slave-8n77 krb5kdc[24387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:10:19.736553 24403 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260429 14:10:19.737135 24403 webserver.cc:492] Webserver started at http://127.0.0.1:43793/ using document root <none> and password file <none>
I20260429 14:10:19.737344 24403 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:19.737404 24403 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:19.737511 24403 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:19.738588 24403 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "5f5da22e71d045caada1734e3f4de6c5"
format_stamp: "Formatted at 2026-04-29 14:10:19 on dist-test-slave-8n77"
server_key: "3e3e138bb0c68c5baf7e6bf9c5f491ba"
server_key_iv: "dc3975355cfc2762cf99ecdd0f4d2fc5"
server_key_version: "encryptionkey@0"
I20260429 14:10:19.738933 24403 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "5f5da22e71d045caada1734e3f4de6c5"
format_stamp: "Formatted at 2026-04-29 14:10:19 on dist-test-slave-8n77"
server_key: "3e3e138bb0c68c5baf7e6bf9c5f491ba"
server_key_iv: "dc3975355cfc2762cf99ecdd0f4d2fc5"
server_key_version: "encryptionkey@0"
I20260429 14:10:19.740309 24403 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:10:19.741111 24418 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:19.741346 24403 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:10:19.741427 24403 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "5f5da22e71d045caada1734e3f4de6c5"
format_stamp: "Formatted at 2026-04-29 14:10:19 on dist-test-slave-8n77"
server_key: "3e3e138bb0c68c5baf7e6bf9c5f491ba"
server_key_iv: "dc3975355cfc2762cf99ecdd0f4d2fc5"
server_key_version: "encryptionkey@0"
I20260429 14:10:19.741513 24403 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:19.745160 24403 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:19.747359 24403 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:19.747619 24403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:19.752058 24403 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45289
I20260429 14:10:19.752110 24470 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:45289 every 8 connection(s)
I20260429 14:10:19.752421 24403 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:10:19.753011 24471 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:19.755563 24471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f5da22e71d045caada1734e3f4de6c5: Bootstrap starting.
I20260429 14:10:19.756152 24471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f5da22e71d045caada1734e3f4de6c5: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:19.756423 24471 log.cc:826] T 00000000000000000000000000000000 P 5f5da22e71d045caada1734e3f4de6c5: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:19.757128 24471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f5da22e71d045caada1734e3f4de6c5: No bootstrap required, opened a new log
I20260429 14:10:19.758462 24471 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5f5da22e71d045caada1734e3f4de6c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f5da22e71d045caada1734e3f4de6c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45289 } }
I20260429 14:10:19.758579 24471 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5f5da22e71d045caada1734e3f4de6c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:10:19.758600 24471 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5f5da22e71d045caada1734e3f4de6c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f5da22e71d045caada1734e3f4de6c5, State: Initialized, Role: FOLLOWER
I20260429 14:10:19.758747 24471 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f5da22e71d045caada1734e3f4de6c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f5da22e71d045caada1734e3f4de6c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45289 } }
I20260429 14:10:19.758801 24471 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5f5da22e71d045caada1734e3f4de6c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:10:19.758823 24471 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5f5da22e71d045caada1734e3f4de6c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:10:19.758847 24471 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5f5da22e71d045caada1734e3f4de6c5 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:10:19.759404 24471 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5f5da22e71d045caada1734e3f4de6c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f5da22e71d045caada1734e3f4de6c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45289 } }
I20260429 14:10:19.759491 24471 leader_election.cc:304] T 00000000000000000000000000000000 P 5f5da22e71d045caada1734e3f4de6c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f5da22e71d045caada1734e3f4de6c5; no voters:
I20260429 14:10:19.759693 24471 leader_election.cc:290] T 00000000000000000000000000000000 P 5f5da22e71d045caada1734e3f4de6c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:10:19.759815 24474 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5f5da22e71d045caada1734e3f4de6c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:10:19.759910 24471 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5f5da22e71d045caada1734e3f4de6c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:10:19.759975 24474 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5f5da22e71d045caada1734e3f4de6c5 [term 1 LEADER]: Becoming Leader. State: Replica: 5f5da22e71d045caada1734e3f4de6c5, State: Running, Role: LEADER
I20260429 14:10:19.760066 24474 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f5da22e71d045caada1734e3f4de6c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f5da22e71d045caada1734e3f4de6c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45289 } }
I20260429 14:10:19.760586 24476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f5da22e71d045caada1734e3f4de6c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f5da22e71d045caada1734e3f4de6c5. Latest consensus state: current_term: 1 leader_uuid: "5f5da22e71d045caada1734e3f4de6c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f5da22e71d045caada1734e3f4de6c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45289 } } }
I20260429 14:10:19.760591 24475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f5da22e71d045caada1734e3f4de6c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f5da22e71d045caada1734e3f4de6c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f5da22e71d045caada1734e3f4de6c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45289 } } }
I20260429 14:10:19.760689 24476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f5da22e71d045caada1734e3f4de6c5 [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:19.760700 24475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f5da22e71d045caada1734e3f4de6c5 [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:19.760892 24483 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:10:19.761514 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 24403
I20260429 14:10:19.761596 24483 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:10:19.761646 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:10:19.761849 2839 external_mini_cluster.cc:1468] Setting key 141439a19aeca671855441d3efdebb90
I20260429 14:10:19.767547 24483 catalog_manager.cc:1357] Generated new cluster ID: 0ce855ef885a4210b423d526d5c8f12b
I20260429 14:10:19.767624 24483 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:10:19.769207 24493 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:19.762330 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:52547 (local address 127.0.0.1:45289)
0429 14:10:19.762469 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:19.762478 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:19.767158 (+ 4680us) server_negotiation.cc:408] Connection header received
0429 14:10:19.767334 (+ 176us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:19.767340 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:19.767365 (+ 25us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:19.767543 (+ 178us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:19.768740 (+ 1197us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:19.768743 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:19.768744 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:10:19.768749 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:19.768775 (+ 26us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:10:19.768790 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:19.768874 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:19.768949 (+ 75us) server_negotiation.cc:300] Negotiation successful
0429 14:10:19.769086 (+ 137us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":77,"threads_started":1}
I20260429 14:10:19.770175 2839 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260429 14:10:19.775755 24493 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:19.771827 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:39917 (local address 127.0.0.1:45289)
0429 14:10:19.771868 (+ 41us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:19.771869 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:19.774534 (+ 2665us) server_negotiation.cc:408] Connection header received
0429 14:10:19.774643 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:19.774644 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:19.774650 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:19.774710 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:19.775385 (+ 675us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:19.775386 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:19.775387 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:10:19.775392 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:19.775407 (+ 15us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:10:19.775411 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:19.775471 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:19.775642 (+ 171us) server_negotiation.cc:300] Negotiation successful
0429 14:10:19.775659 (+ 17us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260429 14:10:19.778469 24483 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:10:19.778645 24483 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:10:19.784677 24483 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 5f5da22e71d045caada1734e3f4de6c5: Generated new TSK 0
I20260429 14:10:19.784881 24483 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 14:10:19.786834 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 24403
2026-04-29T14:10:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2317 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:10:19 dist-test-slave-8n77 krb5kdc[24502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:10:19 dist-test-slave-8n77 krb5kdc[24502](info): set up 2 sockets
Apr 29 14:10:19 dist-test-slave-8n77 krb5kdc[24502](info): commencing operation
krb5kdc: starting...
W20260429 14:10:21.819690 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:10:21 dist-test-slave-8n77 krb5kdc[24502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:10:21Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:10:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:10:21.973902 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34227
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:43085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:10:22.055424 24518 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:22.055647 24518 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:22.055672 24518 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:10:22.057353 24518 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:10:22.057411 24518 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:22.057426 24518 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:22.057439 24518 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:10:22.057451 24518 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:10:22.059382 24518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:22.059963 24518 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:22.060425 24518 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:10:22.063277 24526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:22.063292 24518 server_base.cc:1061] running on GCE node
W20260429 14:10:22.063280 24524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:22.063302 24523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:22.063803 24518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:22.064078 24518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:22.065272 24518 hybrid_clock.cc:648] HybridClock initialized: now 1777471822065248 us; error 32 us; skew 500 ppm
Apr 29 14:10:22 dist-test-slave-8n77 krb5kdc[24502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:10:22.067905 24518 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260429 14:10:22.068681 24518 webserver.cc:492] Webserver started at http://127.0.0.1:41153/ using document root <none> and password file <none>
I20260429 14:10:22.068902 24518 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:22.068964 24518 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:22.069069 24518 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:22.070030 24518 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "8ca0297ce33243248c8d084e40d62826"
format_stamp: "Formatted at 2026-04-29 14:10:22 on dist-test-slave-8n77"
server_key: "8c655a295eba8fc2edf04c5e012f8be0"
server_key_iv: "583512f30801748830d68859e43884b7"
server_key_version: "encryptionkey@0"
I20260429 14:10:22.070448 24518 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "8ca0297ce33243248c8d084e40d62826"
format_stamp: "Formatted at 2026-04-29 14:10:22 on dist-test-slave-8n77"
server_key: "8c655a295eba8fc2edf04c5e012f8be0"
server_key_iv: "583512f30801748830d68859e43884b7"
server_key_version: "encryptionkey@0"
I20260429 14:10:22.071771 24518 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:10:22.072659 24533 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:22.072925 24518 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:10:22.073014 24518 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "8ca0297ce33243248c8d084e40d62826"
format_stamp: "Formatted at 2026-04-29 14:10:22 on dist-test-slave-8n77"
server_key: "8c655a295eba8fc2edf04c5e012f8be0"
server_key_iv: "583512f30801748830d68859e43884b7"
server_key_version: "encryptionkey@0"
I20260429 14:10:22.073096 24518 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:22.094097 24518 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:22.096421 24518 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:22.096608 24518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:22.102164 24585 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:34227 every 8 connection(s)
I20260429 14:10:22.102224 24518 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34227
I20260429 14:10:22.102804 24518 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:10:22.103502 24586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:22.106130 24586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ca0297ce33243248c8d084e40d62826: Bootstrap starting.
I20260429 14:10:22.106844 24586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ca0297ce33243248c8d084e40d62826: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:22.107121 24586 log.cc:826] T 00000000000000000000000000000000 P 8ca0297ce33243248c8d084e40d62826: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:22.107934 24586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ca0297ce33243248c8d084e40d62826: No bootstrap required, opened a new log
I20260429 14:10:22.109292 24586 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8ca0297ce33243248c8d084e40d62826 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ca0297ce33243248c8d084e40d62826" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34227 } }
I20260429 14:10:22.109416 24586 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8ca0297ce33243248c8d084e40d62826 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:10:22.109438 24586 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8ca0297ce33243248c8d084e40d62826 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ca0297ce33243248c8d084e40d62826, State: Initialized, Role: FOLLOWER
I20260429 14:10:22.109566 24586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ca0297ce33243248c8d084e40d62826 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ca0297ce33243248c8d084e40d62826" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34227 } }
I20260429 14:10:22.109660 24586 raft_consensus.cc:399] T 00000000000000000000000000000000 P 8ca0297ce33243248c8d084e40d62826 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:10:22.109643 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 24518
I20260429 14:10:22.109704 24586 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8ca0297ce33243248c8d084e40d62826 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:10:22.109748 24586 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8ca0297ce33243248c8d084e40d62826 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:10:22.109784 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:10:22.109956 2839 external_mini_cluster.cc:1468] Setting key a64f70037490a5e8c7da66742b05a1ca
I20260429 14:10:22.110670 24586 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8ca0297ce33243248c8d084e40d62826 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ca0297ce33243248c8d084e40d62826" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34227 } }
I20260429 14:10:22.110814 24586 leader_election.cc:304] T 00000000000000000000000000000000 P 8ca0297ce33243248c8d084e40d62826 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ca0297ce33243248c8d084e40d62826; no voters:
I20260429 14:10:22.110996 24586 leader_election.cc:290] T 00000000000000000000000000000000 P 8ca0297ce33243248c8d084e40d62826 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:10:22.111045 24591 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8ca0297ce33243248c8d084e40d62826 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:10:22.111217 24591 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8ca0297ce33243248c8d084e40d62826 [term 1 LEADER]: Becoming Leader. State: Replica: 8ca0297ce33243248c8d084e40d62826, State: Running, Role: LEADER
I20260429 14:10:22.111249 24586 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8ca0297ce33243248c8d084e40d62826 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:10:22.111366 24591 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ca0297ce33243248c8d084e40d62826 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ca0297ce33243248c8d084e40d62826" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34227 } }
I20260429 14:10:22.111690 24592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ca0297ce33243248c8d084e40d62826 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ca0297ce33243248c8d084e40d62826. Latest consensus state: current_term: 1 leader_uuid: "8ca0297ce33243248c8d084e40d62826" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ca0297ce33243248c8d084e40d62826" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34227 } } }
I20260429 14:10:22.111781 24592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ca0297ce33243248c8d084e40d62826 [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:22.111936 24593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ca0297ce33243248c8d084e40d62826 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ca0297ce33243248c8d084e40d62826" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ca0297ce33243248c8d084e40d62826" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34227 } } }
I20260429 14:10:22.111974 24595 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:10:22.112514 24595 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:10:22.113185 24593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ca0297ce33243248c8d084e40d62826 [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:22.114549 24595 catalog_manager.cc:1357] Generated new cluster ID: b12afaa513d2469e92e9a59e6d3c0eda
I20260429 14:10:22.114603 24595 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Apr 29 14:10:22 dist-test-slave-8n77 krb5kdc[24502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260429 14:10:22.121263 24590 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:22.110586 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 10.240.0.178:39677 (local address 127.0.0.1:34227)
0429 14:10:22.110761 (+ 175us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:22.110771 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:22.110813 (+ 42us) server_negotiation.cc:408] Connection header received
0429 14:10:22.110941 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:22.110949 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:22.111003 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:22.111150 (+ 147us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:22.112210 (+ 1060us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:22.113796 (+ 1586us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:22.114635 (+ 839us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:22.114863 (+ 228us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:22.117483 (+ 2620us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:22.117510 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:22.117511 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:22.117526 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:22.119561 (+ 2035us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:22.120022 (+ 461us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:22.120025 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:22.120025 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:22.120079 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:22.120295 (+ 216us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:22.120296 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:22.120296 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:22.120538 (+ 242us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:22.120661 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:22.120864 (+ 203us) server_negotiation.cc:300] Negotiation successful
0429 14:10:22.121084 (+ 220us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":98,"threads_started":1}
I20260429 14:10:22.122314 2839 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260429 14:10:22.129647 24595 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:10:22.130453 24595 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:10:22.131462 24590 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:22.123976 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 10.240.0.178:60043 (local address 127.0.0.1:34227)
0429 14:10:22.124038 (+ 62us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:22.124040 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:22.124068 (+ 28us) server_negotiation.cc:408] Connection header received
0429 14:10:22.124130 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:22.124131 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:22.124141 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:22.124202 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:22.124975 (+ 773us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:22.125710 (+ 735us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:22.126419 (+ 709us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:22.126960 (+ 541us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:22.128192 (+ 1232us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:22.128197 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:22.128198 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:22.128216 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:22.130054 (+ 1838us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:22.130527 (+ 473us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:22.130529 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:22.130529 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:22.130554 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:22.130782 (+ 228us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:22.130783 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:22.130783 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:22.130906 (+ 123us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:22.131015 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:22.131285 (+ 270us) server_negotiation.cc:300] Negotiation successful
0429 14:10:22.131353 (+ 68us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260429 14:10:22.137959 24595 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 8ca0297ce33243248c8d084e40d62826: Generated new TSK 0
I20260429 14:10:22.138173 24595 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 14:10:22.144743 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 24518
W20260429 14:10:22.149474 24500 connection.cc:570] client connection to 127.0.0.1:34227 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34227): Connection reset by peer (error 104)
2026-04-29T14:10:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2358 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:10:22 dist-test-slave-8n77 krb5kdc[24617](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:10:22 dist-test-slave-8n77 krb5kdc[24617](info): set up 2 sockets
Apr 29 14:10:22 dist-test-slave-8n77 krb5kdc[24617](info): commencing operation
krb5kdc: starting...
W20260429 14:10:24.195611 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.032s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:10:24 dist-test-slave-8n77 krb5kdc[24617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:10:24Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:10:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:10:24.338989 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35731
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:35831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:10:24.417198 24633 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:24.417443 24633 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:24.417490 24633 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:10:24.419030 24633 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:10:24.419111 24633 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:24.419135 24633 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:24.419157 24633 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:10:24.419188 24633 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:10:24.420831 24633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:24.421096 24633 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:24.421284 24633 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:10:24.423606 24638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:24.423727 24641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:24.423784 24639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:24.423853 24633 server_base.cc:1061] running on GCE node
I20260429 14:10:24.424142 24633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:24.424383 24633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:24.425537 24633 hybrid_clock.cc:648] HybridClock initialized: now 1777471824425513 us; error 37 us; skew 500 ppm
Apr 29 14:10:24 dist-test-slave-8n77 krb5kdc[24617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:10:24.427966 24633 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260429 14:10:24.428550 24633 webserver.cc:492] Webserver started at http://127.0.0.1:44385/ using document root <none> and password file <none>
I20260429 14:10:24.428750 24633 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:24.428808 24633 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:24.428910 24633 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:24.429868 24633 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "12871ff40f0f40659b37248d1dd12b46"
format_stamp: "Formatted at 2026-04-29 14:10:24 on dist-test-slave-8n77"
server_key: "198106209d9025de166e6a2c3b0c84fb"
server_key_iv: "cdac8e2215b95a7854641c83d63861d7"
server_key_version: "encryptionkey@0"
I20260429 14:10:24.430200 24633 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "12871ff40f0f40659b37248d1dd12b46"
format_stamp: "Formatted at 2026-04-29 14:10:24 on dist-test-slave-8n77"
server_key: "198106209d9025de166e6a2c3b0c84fb"
server_key_iv: "cdac8e2215b95a7854641c83d63861d7"
server_key_version: "encryptionkey@0"
I20260429 14:10:24.431452 24633 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:10:24.432196 24648 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:24.432403 24633 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:10:24.432482 24633 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "12871ff40f0f40659b37248d1dd12b46"
format_stamp: "Formatted at 2026-04-29 14:10:24 on dist-test-slave-8n77"
server_key: "198106209d9025de166e6a2c3b0c84fb"
server_key_iv: "cdac8e2215b95a7854641c83d63861d7"
server_key_version: "encryptionkey@0"
I20260429 14:10:24.432569 24633 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:24.452344 24633 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:24.454396 24633 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:24.454560 24633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:24.458408 24633 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35731
I20260429 14:10:24.458462 24700 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:35731 every 8 connection(s)
I20260429 14:10:24.458781 24633 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:10:24.459332 24701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:24.461647 24701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12871ff40f0f40659b37248d1dd12b46: Bootstrap starting.
I20260429 14:10:24.462214 24701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12871ff40f0f40659b37248d1dd12b46: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:24.462502 24701 log.cc:826] T 00000000000000000000000000000000 P 12871ff40f0f40659b37248d1dd12b46: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:24.463169 24701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12871ff40f0f40659b37248d1dd12b46: No bootstrap required, opened a new log
I20260429 14:10:24.463536 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 24633
I20260429 14:10:24.463670 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:10:24.463835 2839 external_mini_cluster.cc:1468] Setting key 33ab2c0ab7ba0ff43c4440061126aed1
I20260429 14:10:24.464776 24701 raft_consensus.cc:359] T 00000000000000000000000000000000 P 12871ff40f0f40659b37248d1dd12b46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12871ff40f0f40659b37248d1dd12b46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35731 } }
I20260429 14:10:24.464928 24701 raft_consensus.cc:385] T 00000000000000000000000000000000 P 12871ff40f0f40659b37248d1dd12b46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:10:24.464962 24701 raft_consensus.cc:740] T 00000000000000000000000000000000 P 12871ff40f0f40659b37248d1dd12b46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12871ff40f0f40659b37248d1dd12b46, State: Initialized, Role: FOLLOWER
I20260429 14:10:24.465070 24701 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12871ff40f0f40659b37248d1dd12b46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12871ff40f0f40659b37248d1dd12b46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35731 } }
I20260429 14:10:24.465131 24701 raft_consensus.cc:399] T 00000000000000000000000000000000 P 12871ff40f0f40659b37248d1dd12b46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:10:24.465158 24701 raft_consensus.cc:493] T 00000000000000000000000000000000 P 12871ff40f0f40659b37248d1dd12b46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:10:24.465197 24701 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 12871ff40f0f40659b37248d1dd12b46 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:10:24.465876 24701 raft_consensus.cc:515] T 00000000000000000000000000000000 P 12871ff40f0f40659b37248d1dd12b46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12871ff40f0f40659b37248d1dd12b46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35731 } }
I20260429 14:10:24.465996 24701 leader_election.cc:304] T 00000000000000000000000000000000 P 12871ff40f0f40659b37248d1dd12b46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12871ff40f0f40659b37248d1dd12b46; no voters:
I20260429 14:10:24.466125 24701 leader_election.cc:290] T 00000000000000000000000000000000 P 12871ff40f0f40659b37248d1dd12b46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:10:24.466254 24706 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 12871ff40f0f40659b37248d1dd12b46 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:10:24.466336 24701 sys_catalog.cc:565] T 00000000000000000000000000000000 P 12871ff40f0f40659b37248d1dd12b46 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:10:24.466495 24706 raft_consensus.cc:697] T 00000000000000000000000000000000 P 12871ff40f0f40659b37248d1dd12b46 [term 1 LEADER]: Becoming Leader. State: Replica: 12871ff40f0f40659b37248d1dd12b46, State: Running, Role: LEADER
I20260429 14:10:24.466625 24706 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12871ff40f0f40659b37248d1dd12b46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12871ff40f0f40659b37248d1dd12b46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35731 } }
I20260429 14:10:24.467015 24708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12871ff40f0f40659b37248d1dd12b46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12871ff40f0f40659b37248d1dd12b46. Latest consensus state: current_term: 1 leader_uuid: "12871ff40f0f40659b37248d1dd12b46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12871ff40f0f40659b37248d1dd12b46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35731 } } }
I20260429 14:10:24.466992 24707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12871ff40f0f40659b37248d1dd12b46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12871ff40f0f40659b37248d1dd12b46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12871ff40f0f40659b37248d1dd12b46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35731 } } }
I20260429 14:10:24.467113 24708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12871ff40f0f40659b37248d1dd12b46 [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:24.467116 24707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12871ff40f0f40659b37248d1dd12b46 [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:24.467502 24715 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:10:24.468169 24715 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:10:24.469120 24705 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:24.464358 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 10.240.0.178:44481 (local address 127.0.0.1:35731)
0429 14:10:24.464523 (+ 165us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:24.464532 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:24.464566 (+ 34us) server_negotiation.cc:408] Connection header received
0429 14:10:24.464734 (+ 168us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:24.464738 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:24.464774 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:24.464857 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:24.465342 (+ 485us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:24.466565 (+ 1223us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:24.467673 (+ 1108us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:24.467945 (+ 272us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:24.468494 (+ 549us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:24.468501 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:24.468502 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:10:24.468702 (+ 200us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:24.468721 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:10:24.468729 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:24.468809 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:24.468948 (+ 139us) server_negotiation.cc:300] Negotiation successful
0429 14:10:24.468971 (+ 23us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":96,"threads_started":1}
I20260429 14:10:24.469646 24715 catalog_manager.cc:1357] Generated new cluster ID: 5c98ed5704c14ecd9d51a29dd4105e19
I20260429 14:10:24.469718 24715 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:10:24.470247 2839 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260429 14:10:24.474587 24705 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:24.471765 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 10.240.0.178:38863 (local address 127.0.0.1:35731)
0429 14:10:24.471803 (+ 38us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:24.471805 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:24.471814 (+ 9us) server_negotiation.cc:408] Connection header received
0429 14:10:24.471872 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:24.471872 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:24.471884 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:24.471930 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:24.472242 (+ 312us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:24.472947 (+ 705us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:24.473640 (+ 693us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:24.473808 (+ 168us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:24.474260 (+ 452us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:24.474263 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:24.474263 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:10:24.474268 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:24.474308 (+ 40us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:10:24.474312 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:24.474383 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:24.474498 (+ 115us) server_negotiation.cc:300] Negotiation successful
0429 14:10:24.474510 (+ 12us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260429 14:10:24.475925 24715 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:10:24.476609 24715 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:10:24.484097 24715 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 12871ff40f0f40659b37248d1dd12b46: Generated new TSK 0
I20260429 14:10:24.484268 24715 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 14:10:24.488014 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 24633
W20260429 14:10:24.492777 24615 connection.cc:570] client connection to 127.0.0.1:35731 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35731): Connection reset by peer (error 104)
2026-04-29T14:10:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2342 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:10:24 dist-test-slave-8n77 krb5kdc[24732](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:10:24 dist-test-slave-8n77 krb5kdc[24732](info): set up 2 sockets
Apr 29 14:10:24 dist-test-slave-8n77 krb5kdc[24732](info): commencing operation
krb5kdc: starting...
W20260429 14:10:26.523970 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:10:26 dist-test-slave-8n77 krb5kdc[24732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:10:26Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:10:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:10:26.674655 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34371
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:44761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:10:26.751786 24748 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:26.752031 24748 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:26.752069 24748 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:10:26.753602 24748 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:10:26.753685 24748 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:26.753715 24748 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:26.753746 24748 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:10:26.753772 24748 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:10:26.755532 24748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:26.755810 24748 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:26.755999 24748 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:10:26.758459 24754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:26.758466 24756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:26.758493 24753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:26.758685 24748 server_base.cc:1061] running on GCE node
I20260429 14:10:26.758867 24748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:26.759112 24748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:26.760281 24748 hybrid_clock.cc:648] HybridClock initialized: now 1777471826760255 us; error 48 us; skew 500 ppm
Apr 29 14:10:26 dist-test-slave-8n77 krb5kdc[24732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:10:26.762741 24748 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260429 14:10:26.763312 24748 webserver.cc:492] Webserver started at http://127.0.0.1:33913/ using document root <none> and password file <none>
I20260429 14:10:26.763520 24748 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:26.763562 24748 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:26.763670 24748 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:26.764607 24748 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "cab38f8db0de42bf8623be5af083b289"
format_stamp: "Formatted at 2026-04-29 14:10:26 on dist-test-slave-8n77"
server_key: "ff5cbc58be061441e5d1ffa37d123db6"
server_key_iv: "550e85b0f9392c9f69044619c56f1350"
server_key_version: "encryptionkey@0"
I20260429 14:10:26.764935 24748 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "cab38f8db0de42bf8623be5af083b289"
format_stamp: "Formatted at 2026-04-29 14:10:26 on dist-test-slave-8n77"
server_key: "ff5cbc58be061441e5d1ffa37d123db6"
server_key_iv: "550e85b0f9392c9f69044619c56f1350"
server_key_version: "encryptionkey@0"
I20260429 14:10:26.766184 24748 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260429 14:10:26.766985 24763 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:26.767273 24748 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:10:26.767366 24748 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "cab38f8db0de42bf8623be5af083b289"
format_stamp: "Formatted at 2026-04-29 14:10:26 on dist-test-slave-8n77"
server_key: "ff5cbc58be061441e5d1ffa37d123db6"
server_key_iv: "550e85b0f9392c9f69044619c56f1350"
server_key_version: "encryptionkey@0"
I20260429 14:10:26.767452 24748 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:26.770839 24748 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:26.772853 24748 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:26.773013 24748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:26.777045 24748 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34371
I20260429 14:10:26.777092 24815 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:34371 every 8 connection(s)
I20260429 14:10:26.777418 24748 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:10:26.777961 24816 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:26.779063 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 24748
I20260429 14:10:26.779165 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:10:26.779371 2839 external_mini_cluster.cc:1468] Setting key d5769672942c3e6bcffbd5895738179c
I20260429 14:10:26.781313 24816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cab38f8db0de42bf8623be5af083b289: Bootstrap starting.
I20260429 14:10:26.781541 24819 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:26.779902 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 10.240.0.178:38325 (local address 127.0.0.1:34371)
0429 14:10:26.780050 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:26.780059 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:26.780076 (+ 17us) server_negotiation.cc:408] Connection header received
0429 14:10:26.780417 (+ 341us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:26.780422 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:26.780445 (+ 23us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:26.780541 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:26.781188 (+ 647us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:26.781190 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:26.781191 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:10:26.781194 (+ 3us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:26.781215 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:10:26.781221 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:26.781293 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:26.781363 (+ 70us) server_negotiation.cc:300] Negotiation successful
0429 14:10:26.781416 (+ 53us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
I20260429 14:10:26.782109 24816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cab38f8db0de42bf8623be5af083b289: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:26.782446 24816 log.cc:826] T 00000000000000000000000000000000 P cab38f8db0de42bf8623be5af083b289: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:26.782761 2839 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260429 14:10:26.783224 24816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cab38f8db0de42bf8623be5af083b289: No bootstrap required, opened a new log
I20260429 14:10:26.784958 24816 raft_consensus.cc:359] T 00000000000000000000000000000000 P cab38f8db0de42bf8623be5af083b289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab38f8db0de42bf8623be5af083b289" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34371 } }
I20260429 14:10:26.785104 24816 raft_consensus.cc:385] T 00000000000000000000000000000000 P cab38f8db0de42bf8623be5af083b289 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:10:26.785135 24816 raft_consensus.cc:740] T 00000000000000000000000000000000 P cab38f8db0de42bf8623be5af083b289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab38f8db0de42bf8623be5af083b289, State: Initialized, Role: FOLLOWER
I20260429 14:10:26.785257 24816 consensus_queue.cc:260] T 00000000000000000000000000000000 P cab38f8db0de42bf8623be5af083b289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab38f8db0de42bf8623be5af083b289" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34371 } }
I20260429 14:10:26.785333 24816 raft_consensus.cc:399] T 00000000000000000000000000000000 P cab38f8db0de42bf8623be5af083b289 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:10:26.785370 24816 raft_consensus.cc:493] T 00000000000000000000000000000000 P cab38f8db0de42bf8623be5af083b289 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:10:26.785424 24816 raft_consensus.cc:3060] T 00000000000000000000000000000000 P cab38f8db0de42bf8623be5af083b289 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:10:26.785627 24819 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:26.784381 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 10.240.0.178:36153 (local address 127.0.0.1:34371)
0429 14:10:26.784439 (+ 58us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:26.784441 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:26.784619 (+ 178us) server_negotiation.cc:408] Connection header received
0429 14:10:26.784641 (+ 22us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:26.784642 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:26.784646 (+ 4us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:26.784691 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:26.785391 (+ 700us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:26.785393 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:26.785393 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:10:26.785397 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:26.785413 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:10:26.785418 (+ 5us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:26.785497 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:26.785529 (+ 32us) server_negotiation.cc:300] Negotiation successful
0429 14:10:26.785541 (+ 12us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260429 14:10:26.786170 24816 raft_consensus.cc:515] T 00000000000000000000000000000000 P cab38f8db0de42bf8623be5af083b289 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab38f8db0de42bf8623be5af083b289" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34371 } }
I20260429 14:10:26.786263 24816 leader_election.cc:304] T 00000000000000000000000000000000 P cab38f8db0de42bf8623be5af083b289 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cab38f8db0de42bf8623be5af083b289; no voters:
I20260429 14:10:26.786451 24816 leader_election.cc:290] T 00000000000000000000000000000000 P cab38f8db0de42bf8623be5af083b289 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:10:26.786499 24826 raft_consensus.cc:2804] T 00000000000000000000000000000000 P cab38f8db0de42bf8623be5af083b289 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:10:26.786648 24826 raft_consensus.cc:697] T 00000000000000000000000000000000 P cab38f8db0de42bf8623be5af083b289 [term 1 LEADER]: Becoming Leader. State: Replica: cab38f8db0de42bf8623be5af083b289, State: Running, Role: LEADER
I20260429 14:10:26.786679 24816 sys_catalog.cc:565] T 00000000000000000000000000000000 P cab38f8db0de42bf8623be5af083b289 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:10:26.786800 24826 consensus_queue.cc:237] T 00000000000000000000000000000000 P cab38f8db0de42bf8623be5af083b289 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab38f8db0de42bf8623be5af083b289" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34371 } }
I20260429 14:10:26.787163 24827 sys_catalog.cc:455] T 00000000000000000000000000000000 P cab38f8db0de42bf8623be5af083b289 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cab38f8db0de42bf8623be5af083b289" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab38f8db0de42bf8623be5af083b289" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34371 } } }
I20260429 14:10:26.787199 24828 sys_catalog.cc:455] T 00000000000000000000000000000000 P cab38f8db0de42bf8623be5af083b289 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cab38f8db0de42bf8623be5af083b289. Latest consensus state: current_term: 1 leader_uuid: "cab38f8db0de42bf8623be5af083b289" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab38f8db0de42bf8623be5af083b289" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34371 } } }
I20260429 14:10:26.787281 24827 sys_catalog.cc:458] T 00000000000000000000000000000000 P cab38f8db0de42bf8623be5af083b289 [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:26.787285 24828 sys_catalog.cc:458] T 00000000000000000000000000000000 P cab38f8db0de42bf8623be5af083b289 [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:26.787637 24834 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:10:26.788084 24834 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:10:26.789443 24834 catalog_manager.cc:1357] Generated new cluster ID: fa81fc2d53ae45a99b426f2cdfb13052
I20260429 14:10:26.789525 24834 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:10:26.796116 24834 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:10:26.796264 24834 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:10:26.801101 24834 catalog_manager.cc:6044] T 00000000000000000000000000000000 P cab38f8db0de42bf8623be5af083b289: Generated new TSK 0
I20260429 14:10:26.801270 24834 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 14:10:26.805642 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 24748
2026-04-29T14:10:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:10:26 dist-test-slave-8n77 krb5kdc[24847](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:10:26 dist-test-slave-8n77 krb5kdc[24847](info): set up 2 sockets
Apr 29 14:10:26 dist-test-slave-8n77 krb5kdc[24847](info): commencing operation
krb5kdc: starting...
W20260429 14:10:28.811595 2839 mini_kdc.cc:121] Time spent starting KDC: real 1.986s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:10:28 dist-test-slave-8n77 krb5kdc[24847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:10:28Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:10:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:10:28.957307 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42811
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:41461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:10:29.033939 24863 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:29.034137 24863 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:29.034160 24863 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:10:29.035707 24863 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:10:29.035756 24863 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:29.035771 24863 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:29.035784 24863 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:10:29.035797 24863 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:10:29.037415 24863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42811
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:29.037650 24863 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:29.037829 24863 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:10:29.040482 24869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:29.040514 24871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:29.040547 24868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:29.040635 24863 server_base.cc:1061] running on GCE node
I20260429 14:10:29.040831 24863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:29.041074 24863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:29.042236 24863 hybrid_clock.cc:648] HybridClock initialized: now 1777471829042209 us; error 49 us; skew 500 ppm
Apr 29 14:10:29 dist-test-slave-8n77 krb5kdc[24847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:10:29.044551 24863 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260429 14:10:29.045037 24863 webserver.cc:492] Webserver started at http://127.0.0.1:45841/ using document root <none> and password file <none>
I20260429 14:10:29.045229 24863 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:29.045288 24863 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:29.045387 24863 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:29.046325 24863 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "1993885209864e42bb713dbb7257dc01"
format_stamp: "Formatted at 2026-04-29 14:10:29 on dist-test-slave-8n77"
server_key: "b5fdff1cab18f0eaa0739f4a0294d1f9"
server_key_iv: "0e161d81c691bea3c40a5a35ec1f6aa4"
server_key_version: "encryptionkey@0"
I20260429 14:10:29.046657 24863 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "1993885209864e42bb713dbb7257dc01"
format_stamp: "Formatted at 2026-04-29 14:10:29 on dist-test-slave-8n77"
server_key: "b5fdff1cab18f0eaa0739f4a0294d1f9"
server_key_iv: "0e161d81c691bea3c40a5a35ec1f6aa4"
server_key_version: "encryptionkey@0"
I20260429 14:10:29.047832 24863 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:10:29.048506 24878 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:29.048694 24863 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260429 14:10:29.048779 24863 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "1993885209864e42bb713dbb7257dc01"
format_stamp: "Formatted at 2026-04-29 14:10:29 on dist-test-slave-8n77"
server_key: "b5fdff1cab18f0eaa0739f4a0294d1f9"
server_key_iv: "0e161d81c691bea3c40a5a35ec1f6aa4"
server_key_version: "encryptionkey@0"
I20260429 14:10:29.048854 24863 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:29.069754 24863 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:29.079800 24863 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:29.079973 24863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:29.083948 24863 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42811
I20260429 14:10:29.083997 24930 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:42811 every 8 connection(s)
I20260429 14:10:29.084328 24863 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:10:29.084945 24931 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:29.087337 24931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1993885209864e42bb713dbb7257dc01: Bootstrap starting.
I20260429 14:10:29.087881 24931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1993885209864e42bb713dbb7257dc01: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:29.088076 24931 log.cc:826] T 00000000000000000000000000000000 P 1993885209864e42bb713dbb7257dc01: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:29.088734 24931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1993885209864e42bb713dbb7257dc01: No bootstrap required, opened a new log
I20260429 14:10:29.089849 24931 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1993885209864e42bb713dbb7257dc01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1993885209864e42bb713dbb7257dc01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42811 } }
I20260429 14:10:29.089962 24931 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1993885209864e42bb713dbb7257dc01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:10:29.089983 24931 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1993885209864e42bb713dbb7257dc01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1993885209864e42bb713dbb7257dc01, State: Initialized, Role: FOLLOWER
I20260429 14:10:29.090101 24931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1993885209864e42bb713dbb7257dc01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1993885209864e42bb713dbb7257dc01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42811 } }
I20260429 14:10:29.090170 24931 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1993885209864e42bb713dbb7257dc01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:10:29.090198 24931 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1993885209864e42bb713dbb7257dc01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:10:29.090222 24931 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1993885209864e42bb713dbb7257dc01 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:10:29.090771 24931 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1993885209864e42bb713dbb7257dc01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1993885209864e42bb713dbb7257dc01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42811 } }
I20260429 14:10:29.090853 24931 leader_election.cc:304] T 00000000000000000000000000000000 P 1993885209864e42bb713dbb7257dc01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1993885209864e42bb713dbb7257dc01; no voters:
I20260429 14:10:29.090960 24931 leader_election.cc:290] T 00000000000000000000000000000000 P 1993885209864e42bb713dbb7257dc01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:10:29.091056 24934 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1993885209864e42bb713dbb7257dc01 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:10:29.091236 24931 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1993885209864e42bb713dbb7257dc01 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:10:29.091296 24934 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1993885209864e42bb713dbb7257dc01 [term 1 LEADER]: Becoming Leader. State: Replica: 1993885209864e42bb713dbb7257dc01, State: Running, Role: LEADER
I20260429 14:10:29.091410 24934 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1993885209864e42bb713dbb7257dc01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1993885209864e42bb713dbb7257dc01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42811 } }
I20260429 14:10:29.091786 24936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1993885209864e42bb713dbb7257dc01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1993885209864e42bb713dbb7257dc01. Latest consensus state: current_term: 1 leader_uuid: "1993885209864e42bb713dbb7257dc01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1993885209864e42bb713dbb7257dc01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42811 } } }
I20260429 14:10:29.091790 24935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1993885209864e42bb713dbb7257dc01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1993885209864e42bb713dbb7257dc01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1993885209864e42bb713dbb7257dc01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42811 } } }
I20260429 14:10:29.091904 24936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1993885209864e42bb713dbb7257dc01 [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:29.091926 24935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1993885209864e42bb713dbb7257dc01 [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:29.092317 24943 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:10:29.092346 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 24863
I20260429 14:10:29.092480 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:10:29.092679 2839 external_mini_cluster.cc:1468] Setting key 9fd7d5368132dac08a59b56028befbd3
I20260429 14:10:29.092821 24943 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:10:29.094725 24943 catalog_manager.cc:1357] Generated new cluster ID: 5e20b69846974afdbbb8b0f92deb1596
I20260429 14:10:29.094792 24943 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Apr 29 14:10:29 dist-test-slave-8n77 krb5kdc[24847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260429 14:10:29.100347 24943 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:10:29.101027 24943 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:10:29.103175 24953 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:29.093281 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 10.240.0.178:35305 (local address 127.0.0.1:42811)
0429 14:10:29.093419 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:29.093443 (+ 24us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:29.093504 (+ 61us) server_negotiation.cc:408] Connection header received
0429 14:10:29.093619 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:29.093625 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:29.093672 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:29.093788 (+ 116us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:29.094728 (+ 940us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:29.095873 (+ 1145us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:29.096858 (+ 985us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:29.097078 (+ 220us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:29.099612 (+ 2534us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:29.099635 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:29.099635 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:29.099648 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:29.101684 (+ 2036us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:29.102113 (+ 429us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:29.102116 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:29.102116 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:29.102165 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:29.102404 (+ 239us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:29.102405 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:29.102405 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:29.102595 (+ 190us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:29.102743 (+ 148us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:29.102915 (+ 172us) server_negotiation.cc:300] Negotiation successful
0429 14:10:29.103017 (+ 102us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
I20260429 14:10:29.104071 2839 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260429 14:10:29.105177 24943 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 1993885209864e42bb713dbb7257dc01: Generated new TSK 0
I20260429 14:10:29.105330 24943 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 14:10:29.111622 24953 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:29.105594 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 10.240.0.178:60815 (local address 127.0.0.1:42811)
0429 14:10:29.105632 (+ 38us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:29.105634 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:29.105671 (+ 37us) server_negotiation.cc:408] Connection header received
0429 14:10:29.105744 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:29.105745 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:29.105755 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:29.105840 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:29.106517 (+ 677us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:29.107023 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:29.107580 (+ 557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:29.107795 (+ 215us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:29.108845 (+ 1050us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:29.108849 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:29.108849 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:29.108856 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:29.110434 (+ 1578us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:29.110823 (+ 389us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:29.110824 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:29.110825 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:29.110858 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:29.111074 (+ 216us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:29.111074 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:29.111075 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:29.111172 (+ 97us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:29.111294 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:29.111450 (+ 156us) server_negotiation.cc:300] Negotiation successful
0429 14:10:29.111523 (+ 73us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260429 14:10:29.113570 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 24863
W20260429 14:10:29.117610 24845 connection.cc:570] client connection to 127.0.0.1:42811 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42811): Connection reset by peer (error 104)
2026-04-29T14:10:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2307 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:10:29 dist-test-slave-8n77 krb5kdc[24962](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:10:29 dist-test-slave-8n77 krb5kdc[24962](info): set up 2 sockets
Apr 29 14:10:29 dist-test-slave-8n77 krb5kdc[24962](info): commencing operation
krb5kdc: starting...
W20260429 14:10:31.152181 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:10:31 dist-test-slave-8n77 krb5kdc[24962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:10:31Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:10:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:10:31.303021 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42871
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:38385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:10:31.384467 24978 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:31.384722 24978 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:31.384761 24978 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:10:31.386394 24978 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:10:31.386468 24978 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:31.386497 24978 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:31.386524 24978 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:10:31.386543 24978 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:10:31.388507 24978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42871
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:31.388782 24978 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:31.388969 24978 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:10:31.391741 24984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:31.391743 24986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:31.391777 24978 server_base.cc:1061] running on GCE node
W20260429 14:10:31.391741 24983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:31.392202 24978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:31.392405 24978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:31.393599 24978 hybrid_clock.cc:648] HybridClock initialized: now 1777471831393570 us; error 48 us; skew 500 ppm
Apr 29 14:10:31 dist-test-slave-8n77 krb5kdc[24962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:10:31.396085 24978 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260429 14:10:31.396725 24978 webserver.cc:492] Webserver started at http://127.0.0.1:33979/ using document root <none> and password file <none>
I20260429 14:10:31.396941 24978 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:31.396986 24978 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:31.397084 24978 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:31.398056 24978 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "31fa082aa45540ff988fcd311ad2b6cb"
format_stamp: "Formatted at 2026-04-29 14:10:31 on dist-test-slave-8n77"
server_key: "3200977be158ce22170fabf1f14605b9"
server_key_iv: "79ded539b5ae46d5e5f257cfe784dad4"
server_key_version: "encryptionkey@0"
I20260429 14:10:31.398557 24978 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "31fa082aa45540ff988fcd311ad2b6cb"
format_stamp: "Formatted at 2026-04-29 14:10:31 on dist-test-slave-8n77"
server_key: "3200977be158ce22170fabf1f14605b9"
server_key_iv: "79ded539b5ae46d5e5f257cfe784dad4"
server_key_version: "encryptionkey@0"
I20260429 14:10:31.399909 24978 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:10:31.400900 24993 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:31.401086 24978 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:10:31.401197 24978 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "31fa082aa45540ff988fcd311ad2b6cb"
format_stamp: "Formatted at 2026-04-29 14:10:31 on dist-test-slave-8n77"
server_key: "3200977be158ce22170fabf1f14605b9"
server_key_iv: "79ded539b5ae46d5e5f257cfe784dad4"
server_key_version: "encryptionkey@0"
I20260429 14:10:31.401268 24978 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:31.417097 24978 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:31.419174 24978 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:31.419369 24978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:31.424492 24978 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42871
I20260429 14:10:31.424531 25045 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:42871 every 8 connection(s)
I20260429 14:10:31.424865 24978 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:10:31.425522 25046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:31.428027 25046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31fa082aa45540ff988fcd311ad2b6cb: Bootstrap starting.
I20260429 14:10:31.428406 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 24978
I20260429 14:10:31.428541 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:10:31.428730 2839 external_mini_cluster.cc:1468] Setting key 182abd51cb72e4083d2581dbdb6c2f93
I20260429 14:10:31.428740 25046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31fa082aa45540ff988fcd311ad2b6cb: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:31.428987 25046 log.cc:826] T 00000000000000000000000000000000 P 31fa082aa45540ff988fcd311ad2b6cb: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:31.429807 25046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31fa082aa45540ff988fcd311ad2b6cb: No bootstrap required, opened a new log
I20260429 14:10:31.430968 25046 raft_consensus.cc:359] T 00000000000000000000000000000000 P 31fa082aa45540ff988fcd311ad2b6cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31fa082aa45540ff988fcd311ad2b6cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42871 } }
I20260429 14:10:31.431088 25046 raft_consensus.cc:385] T 00000000000000000000000000000000 P 31fa082aa45540ff988fcd311ad2b6cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:10:31.431109 25046 raft_consensus.cc:740] T 00000000000000000000000000000000 P 31fa082aa45540ff988fcd311ad2b6cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31fa082aa45540ff988fcd311ad2b6cb, State: Initialized, Role: FOLLOWER
I20260429 14:10:31.431226 25046 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31fa082aa45540ff988fcd311ad2b6cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31fa082aa45540ff988fcd311ad2b6cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42871 } }
I20260429 14:10:31.431290 25046 raft_consensus.cc:399] T 00000000000000000000000000000000 P 31fa082aa45540ff988fcd311ad2b6cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:10:31.431313 25046 raft_consensus.cc:493] T 00000000000000000000000000000000 P 31fa082aa45540ff988fcd311ad2b6cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:10:31.431378 25046 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 31fa082aa45540ff988fcd311ad2b6cb [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:10:31.431967 25046 raft_consensus.cc:515] T 00000000000000000000000000000000 P 31fa082aa45540ff988fcd311ad2b6cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31fa082aa45540ff988fcd311ad2b6cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42871 } }
I20260429 14:10:31.432215 25046 leader_election.cc:304] T 00000000000000000000000000000000 P 31fa082aa45540ff988fcd311ad2b6cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31fa082aa45540ff988fcd311ad2b6cb; no voters:
I20260429 14:10:31.432427 25046 leader_election.cc:290] T 00000000000000000000000000000000 P 31fa082aa45540ff988fcd311ad2b6cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:10:31.432478 25051 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 31fa082aa45540ff988fcd311ad2b6cb [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:10:31.432596 25051 raft_consensus.cc:697] T 00000000000000000000000000000000 P 31fa082aa45540ff988fcd311ad2b6cb [term 1 LEADER]: Becoming Leader. State: Replica: 31fa082aa45540ff988fcd311ad2b6cb, State: Running, Role: LEADER
I20260429 14:10:31.432709 25046 sys_catalog.cc:565] T 00000000000000000000000000000000 P 31fa082aa45540ff988fcd311ad2b6cb [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:10:31.432753 25051 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31fa082aa45540ff988fcd311ad2b6cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31fa082aa45540ff988fcd311ad2b6cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42871 } }
I20260429 14:10:31.433462 25052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31fa082aa45540ff988fcd311ad2b6cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31fa082aa45540ff988fcd311ad2b6cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31fa082aa45540ff988fcd311ad2b6cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42871 } } }
I20260429 14:10:31.433919 25052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31fa082aa45540ff988fcd311ad2b6cb [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:31.433825 25053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31fa082aa45540ff988fcd311ad2b6cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31fa082aa45540ff988fcd311ad2b6cb. Latest consensus state: current_term: 1 leader_uuid: "31fa082aa45540ff988fcd311ad2b6cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31fa082aa45540ff988fcd311ad2b6cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42871 } } }
I20260429 14:10:31.434151 25050 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:31.429178 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 10.240.0.178:38017 (local address 127.0.0.1:42871)
0429 14:10:31.429337 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:31.429350 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:31.429377 (+ 27us) server_negotiation.cc:408] Connection header received
0429 14:10:31.429955 (+ 578us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:31.429966 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:31.430021 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:31.430150 (+ 129us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:31.430846 (+ 696us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:31.431775 (+ 929us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:31.432732 (+ 957us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:31.432942 (+ 210us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:31.433362 (+ 420us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:31.433370 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:31.433370 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:10:31.433604 (+ 234us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:31.433625 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:10:31.433635 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:31.433737 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:31.433937 (+ 200us) server_negotiation.cc:300] Negotiation successful
0429 14:10:31.433964 (+ 27us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
I20260429 14:10:31.434199 25061 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:10:31.434212 25053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31fa082aa45540ff988fcd311ad2b6cb [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:31.435057 25061 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:10:31.435329 2839 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260429 14:10:31.436828 25061 catalog_manager.cc:1357] Generated new cluster ID: 16345067eaf743088fa96d62cb9e2644
I20260429 14:10:31.436885 25061 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:10:31.439913 25050 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:31.436794 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 10.240.0.178:38027 (local address 127.0.0.1:42871)
0429 14:10:31.436829 (+ 35us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:31.436831 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:31.437114 (+ 283us) server_negotiation.cc:408] Connection header received
0429 14:10:31.437160 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:31.437161 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:31.437172 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:31.437235 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:31.437568 (+ 333us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:31.438257 (+ 689us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:31.439026 (+ 769us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:31.439230 (+ 204us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:31.439641 (+ 411us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:31.439644 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:31.439645 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:10:31.439650 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:31.439664 (+ 14us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:10:31.439668 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:31.439746 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:31.439825 (+ 79us) server_negotiation.cc:300] Negotiation successful
0429 14:10:31.439835 (+ 10us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260429 14:10:31.441926 25061 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:10:31.442515 25061 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:10:31.450433 25061 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 31fa082aa45540ff988fcd311ad2b6cb: Generated new TSK 0
I20260429 14:10:31.450652 25061 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 14:10:31.456184 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 24978
W20260429 14:10:31.460831 24960 connection.cc:570] client connection to 127.0.0.1:42871 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42871): Connection reset by peer (error 104)
2026-04-29T14:10:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2343 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:10:31 dist-test-slave-8n77 krb5kdc[25077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:10:31 dist-test-slave-8n77 krb5kdc[25077](info): set up 2 sockets
Apr 29 14:10:31 dist-test-slave-8n77 krb5kdc[25077](info): commencing operation
krb5kdc: starting...
W20260429 14:10:33.483771 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:10:33 dist-test-slave-8n77 krb5kdc[25077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:10:33Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:10:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:10:33.646265 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44763
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:43165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:10:33.723089 25093 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:33.723306 25093 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:33.723332 25093 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:10:33.724805 25093 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:10:33.724854 25093 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:33.724871 25093 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:33.724885 25093 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:10:33.724898 25093 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:10:33.726583 25093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44763
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:33.726776 25093 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:33.726912 25093 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:10:33.729387 25101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:33.729434 25098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:33.729501 25099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:33.729645 25093 server_base.cc:1061] running on GCE node
I20260429 14:10:33.729849 25093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:33.730100 25093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:33.731271 25093 hybrid_clock.cc:648] HybridClock initialized: now 1777471833731275 us; error 52 us; skew 500 ppm
Apr 29 14:10:33 dist-test-slave-8n77 krb5kdc[25077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:10:33.733633 25093 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260429 14:10:33.734150 25093 webserver.cc:492] Webserver started at http://127.0.0.1:32781/ using document root <none> and password file <none>
I20260429 14:10:33.734375 25093 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:33.734418 25093 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:33.734495 25093 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:33.735458 25093 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "19ce8789e1aa4ac38b17cbb0b1b85040"
format_stamp: "Formatted at 2026-04-29 14:10:33 on dist-test-slave-8n77"
server_key: "db3add55899a77a2530c2eca23f57fe0"
server_key_iv: "c7f7976f2e2fe9de00a200e860a72c96"
server_key_version: "encryptionkey@0"
I20260429 14:10:33.735785 25093 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "19ce8789e1aa4ac38b17cbb0b1b85040"
format_stamp: "Formatted at 2026-04-29 14:10:33 on dist-test-slave-8n77"
server_key: "db3add55899a77a2530c2eca23f57fe0"
server_key_iv: "c7f7976f2e2fe9de00a200e860a72c96"
server_key_version: "encryptionkey@0"
I20260429 14:10:33.737088 25093 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260429 14:10:33.737947 25108 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:33.738178 25093 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:10:33.738246 25093 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "19ce8789e1aa4ac38b17cbb0b1b85040"
format_stamp: "Formatted at 2026-04-29 14:10:33 on dist-test-slave-8n77"
server_key: "db3add55899a77a2530c2eca23f57fe0"
server_key_iv: "c7f7976f2e2fe9de00a200e860a72c96"
server_key_version: "encryptionkey@0"
I20260429 14:10:33.738358 25093 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:33.741590 25093 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:33.743680 25093 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:33.743824 25093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:33.747855 25093 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44763
I20260429 14:10:33.747924 25160 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:44763 every 8 connection(s)
I20260429 14:10:33.748298 25093 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:10:33.748971 25161 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:33.751183 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 25093
I20260429 14:10:33.751341 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:10:33.751540 2839 external_mini_cluster.cc:1468] Setting key f110f77fa3b05d88792604e009df55ca
I20260429 14:10:33.751641 25161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19ce8789e1aa4ac38b17cbb0b1b85040: Bootstrap starting.
I20260429 14:10:33.752524 25161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19ce8789e1aa4ac38b17cbb0b1b85040: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:33.752805 25161 log.cc:826] T 00000000000000000000000000000000 P 19ce8789e1aa4ac38b17cbb0b1b85040: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:33.753564 25161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19ce8789e1aa4ac38b17cbb0b1b85040: No bootstrap required, opened a new log
I20260429 14:10:33.754153 25164 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:33.752120 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 10.240.0.178:48975 (local address 127.0.0.1:44763)
0429 14:10:33.752310 (+ 190us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:33.752321 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:33.752343 (+ 22us) server_negotiation.cc:408] Connection header received
0429 14:10:33.752777 (+ 434us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:33.752787 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:33.752818 (+ 31us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:33.752951 (+ 133us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:33.753825 (+ 874us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:33.753833 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:33.753833 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:10:33.753838 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:33.753859 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:10:33.753864 (+ 5us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:33.753944 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:33.754001 (+ 57us) server_negotiation.cc:300] Negotiation successful
0429 14:10:33.754037 (+ 36us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":91,"threads_started":1}
I20260429 14:10:33.755025 2839 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260429 14:10:33.755378 25161 raft_consensus.cc:359] T 00000000000000000000000000000000 P 19ce8789e1aa4ac38b17cbb0b1b85040 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19ce8789e1aa4ac38b17cbb0b1b85040" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44763 } }
I20260429 14:10:33.755529 25161 raft_consensus.cc:385] T 00000000000000000000000000000000 P 19ce8789e1aa4ac38b17cbb0b1b85040 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:10:33.755577 25161 raft_consensus.cc:740] T 00000000000000000000000000000000 P 19ce8789e1aa4ac38b17cbb0b1b85040 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19ce8789e1aa4ac38b17cbb0b1b85040, State: Initialized, Role: FOLLOWER
I20260429 14:10:33.755699 25161 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19ce8789e1aa4ac38b17cbb0b1b85040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19ce8789e1aa4ac38b17cbb0b1b85040" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44763 } }
I20260429 14:10:33.755761 25161 raft_consensus.cc:399] T 00000000000000000000000000000000 P 19ce8789e1aa4ac38b17cbb0b1b85040 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:10:33.755782 25161 raft_consensus.cc:493] T 00000000000000000000000000000000 P 19ce8789e1aa4ac38b17cbb0b1b85040 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:10:33.755843 25161 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 19ce8789e1aa4ac38b17cbb0b1b85040 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:10:33.756628 25161 raft_consensus.cc:515] T 00000000000000000000000000000000 P 19ce8789e1aa4ac38b17cbb0b1b85040 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19ce8789e1aa4ac38b17cbb0b1b85040" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44763 } }
I20260429 14:10:33.756765 25161 leader_election.cc:304] T 00000000000000000000000000000000 P 19ce8789e1aa4ac38b17cbb0b1b85040 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19ce8789e1aa4ac38b17cbb0b1b85040; no voters:
I20260429 14:10:33.756943 25161 leader_election.cc:290] T 00000000000000000000000000000000 P 19ce8789e1aa4ac38b17cbb0b1b85040 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:10:33.757022 25171 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 19ce8789e1aa4ac38b17cbb0b1b85040 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:10:33.757169 25171 raft_consensus.cc:697] T 00000000000000000000000000000000 P 19ce8789e1aa4ac38b17cbb0b1b85040 [term 1 LEADER]: Becoming Leader. State: Replica: 19ce8789e1aa4ac38b17cbb0b1b85040, State: Running, Role: LEADER
I20260429 14:10:33.757212 25161 sys_catalog.cc:565] T 00000000000000000000000000000000 P 19ce8789e1aa4ac38b17cbb0b1b85040 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:10:33.757874 25164 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:33.756643 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 10.240.0.178:34509 (local address 127.0.0.1:44763)
0429 14:10:33.756704 (+ 61us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:33.756707 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:33.756808 (+ 101us) server_negotiation.cc:408] Connection header received
0429 14:10:33.756898 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:33.756899 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:33.756914 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:33.756982 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:33.757633 (+ 651us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:33.757635 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:33.757635 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:10:33.757639 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:33.757653 (+ 14us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:10:33.757656 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:33.757720 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:33.757778 (+ 58us) server_negotiation.cc:300] Negotiation successful
0429 14:10:33.757790 (+ 12us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260429 14:10:33.758481 25171 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19ce8789e1aa4ac38b17cbb0b1b85040 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19ce8789e1aa4ac38b17cbb0b1b85040" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44763 } }
I20260429 14:10:33.759042 25173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19ce8789e1aa4ac38b17cbb0b1b85040 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19ce8789e1aa4ac38b17cbb0b1b85040. Latest consensus state: current_term: 1 leader_uuid: "19ce8789e1aa4ac38b17cbb0b1b85040" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19ce8789e1aa4ac38b17cbb0b1b85040" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44763 } } }
I20260429 14:10:33.759123 25173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19ce8789e1aa4ac38b17cbb0b1b85040 [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:33.759300 25172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19ce8789e1aa4ac38b17cbb0b1b85040 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19ce8789e1aa4ac38b17cbb0b1b85040" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19ce8789e1aa4ac38b17cbb0b1b85040" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44763 } } }
I20260429 14:10:33.759371 25172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19ce8789e1aa4ac38b17cbb0b1b85040 [sys.catalog]: This master's current role is: LEADER
W20260429 14:10:33.759552 25185 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 19ce8789e1aa4ac38b17cbb0b1b85040: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260429 14:10:33.759604 25185 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260429 14:10:33.759661 25187 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:10:33.759833 25187 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:10:33.761169 25187 catalog_manager.cc:1357] Generated new cluster ID: 860b8dbfdd1f4cfea7c204f597b5e1bc
I20260429 14:10:33.761219 25187 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:10:33.767544 25187 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:10:33.767719 25187 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:10:33.778038 25187 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 19ce8789e1aa4ac38b17cbb0b1b85040: Generated new TSK 0
I20260429 14:10:33.778216 25187 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 14:10:33.784176 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 25093
2026-04-29T14:10:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:10:33 dist-test-slave-8n77 krb5kdc[25192](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:10:33 dist-test-slave-8n77 krb5kdc[25192](info): set up 2 sockets
Apr 29 14:10:33 dist-test-slave-8n77 krb5kdc[25192](info): commencing operation
krb5kdc: starting...
W20260429 14:10:35.803774 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.001s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:10:35 dist-test-slave-8n77 krb5kdc[25192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:10:35Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:10:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:10:35.950052 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:36665
--webserver_interface=127.2.197.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:36743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:36665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:10:36.025653 25208 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:36.025906 25208 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:36.025945 25208 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:10:36.027465 25208 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:10:36.027552 25208 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:36.027576 25208 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:36.027597 25208 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:10:36.027629 25208 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:10:36.029258 25208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:36743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:36665
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:36665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:36.029531 25208 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:36.029717 25208 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:10:36.032079 25213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:36.032081 25214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:36.032119 25216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:36.032414 25208 server_base.cc:1061] running on GCE node
I20260429 14:10:36.032656 25208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:36.032900 25208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:36.034058 25208 hybrid_clock.cc:648] HybridClock initialized: now 1777471836034043 us; error 49 us; skew 500 ppm
Apr 29 14:10:36 dist-test-slave-8n77 krb5kdc[25192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471836, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:10:36.036540 25208 init.cc:377] Logged in from keytab as kudu/127.2.197.254@KRBTEST.COM (short username kudu)
I20260429 14:10:36.037046 25208 webserver.cc:492] Webserver started at http://127.2.197.254:44815/ using document root <none> and password file <none>
I20260429 14:10:36.037242 25208 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:36.037302 25208 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:36.037401 25208 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:36.038364 25208 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "606bddb39aeb4393b0363d81c7ac591a"
format_stamp: "Formatted at 2026-04-29 14:10:36 on dist-test-slave-8n77"
server_key: "e6993dc6850dbe86ab1ad7696e6d0fa5"
server_key_iv: "47dbca32db251158171a59ab00396b55"
server_key_version: "encryptionkey@0"
I20260429 14:10:36.038687 25208 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "606bddb39aeb4393b0363d81c7ac591a"
format_stamp: "Formatted at 2026-04-29 14:10:36 on dist-test-slave-8n77"
server_key: "e6993dc6850dbe86ab1ad7696e6d0fa5"
server_key_iv: "47dbca32db251158171a59ab00396b55"
server_key_version: "encryptionkey@0"
I20260429 14:10:36.039840 25208 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:10:36.040643 25223 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:36.040901 25208 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:10:36.040985 25208 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "606bddb39aeb4393b0363d81c7ac591a"
format_stamp: "Formatted at 2026-04-29 14:10:36 on dist-test-slave-8n77"
server_key: "e6993dc6850dbe86ab1ad7696e6d0fa5"
server_key_iv: "47dbca32db251158171a59ab00396b55"
server_key_version: "encryptionkey@0"
I20260429 14:10:36.041065 25208 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:36.062510 25208 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:36.070567 25208 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:36.070744 25208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:36.075501 25208 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:36665
I20260429 14:10:36.075595 25275 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:36665 every 8 connection(s)
I20260429 14:10:36.075908 25208 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:10:36.076460 25276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:36.078835 25276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 606bddb39aeb4393b0363d81c7ac591a: Bootstrap starting.
I20260429 14:10:36.079437 25276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 606bddb39aeb4393b0363d81c7ac591a: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:36.079706 25276 log.cc:826] T 00000000000000000000000000000000 P 606bddb39aeb4393b0363d81c7ac591a: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:36.080417 25276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 606bddb39aeb4393b0363d81c7ac591a: No bootstrap required, opened a new log
I20260429 14:10:36.081646 25276 raft_consensus.cc:359] T 00000000000000000000000000000000 P 606bddb39aeb4393b0363d81c7ac591a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "606bddb39aeb4393b0363d81c7ac591a" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 36665 } }
I20260429 14:10:36.081759 25276 raft_consensus.cc:385] T 00000000000000000000000000000000 P 606bddb39aeb4393b0363d81c7ac591a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:10:36.081780 25276 raft_consensus.cc:740] T 00000000000000000000000000000000 P 606bddb39aeb4393b0363d81c7ac591a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606bddb39aeb4393b0363d81c7ac591a, State: Initialized, Role: FOLLOWER
I20260429 14:10:36.081938 25276 consensus_queue.cc:260] T 00000000000000000000000000000000 P 606bddb39aeb4393b0363d81c7ac591a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "606bddb39aeb4393b0363d81c7ac591a" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 36665 } }
I20260429 14:10:36.081991 25276 raft_consensus.cc:399] T 00000000000000000000000000000000 P 606bddb39aeb4393b0363d81c7ac591a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:10:36.082011 25276 raft_consensus.cc:493] T 00000000000000000000000000000000 P 606bddb39aeb4393b0363d81c7ac591a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:10:36.082060 25276 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 606bddb39aeb4393b0363d81c7ac591a [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:10:36.082607 25276 raft_consensus.cc:515] T 00000000000000000000000000000000 P 606bddb39aeb4393b0363d81c7ac591a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "606bddb39aeb4393b0363d81c7ac591a" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 36665 } }
I20260429 14:10:36.082724 25276 leader_election.cc:304] T 00000000000000000000000000000000 P 606bddb39aeb4393b0363d81c7ac591a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 606bddb39aeb4393b0363d81c7ac591a; no voters:
I20260429 14:10:36.082892 25276 leader_election.cc:290] T 00000000000000000000000000000000 P 606bddb39aeb4393b0363d81c7ac591a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:10:36.083022 25279 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 606bddb39aeb4393b0363d81c7ac591a [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:10:36.083115 25276 sys_catalog.cc:565] T 00000000000000000000000000000000 P 606bddb39aeb4393b0363d81c7ac591a [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:10:36.083175 25279 raft_consensus.cc:697] T 00000000000000000000000000000000 P 606bddb39aeb4393b0363d81c7ac591a [term 1 LEADER]: Becoming Leader. State: Replica: 606bddb39aeb4393b0363d81c7ac591a, State: Running, Role: LEADER
I20260429 14:10:36.083261 25279 consensus_queue.cc:237] T 00000000000000000000000000000000 P 606bddb39aeb4393b0363d81c7ac591a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "606bddb39aeb4393b0363d81c7ac591a" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 36665 } }
I20260429 14:10:36.083632 25280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 606bddb39aeb4393b0363d81c7ac591a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "606bddb39aeb4393b0363d81c7ac591a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "606bddb39aeb4393b0363d81c7ac591a" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 36665 } } }
I20260429 14:10:36.083725 25280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 606bddb39aeb4393b0363d81c7ac591a [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:36.083875 25281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 606bddb39aeb4393b0363d81c7ac591a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 606bddb39aeb4393b0363d81c7ac591a. Latest consensus state: current_term: 1 leader_uuid: "606bddb39aeb4393b0363d81c7ac591a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "606bddb39aeb4393b0363d81c7ac591a" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 36665 } } }
I20260429 14:10:36.083968 25281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 606bddb39aeb4393b0363d81c7ac591a [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:36.083909 25287 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:10:36.084415 25287 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:10:36.084889 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 25208
I20260429 14:10:36.085011 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:10:36.085175 2839 external_mini_cluster.cc:1468] Setting key ccb317ecaf2794ac8130fd434447258f
I20260429 14:10:36.086517 25287 catalog_manager.cc:1357] Generated new cluster ID: 05c70a9818c14493875fffb1a4f9186b
I20260429 14:10:36.086576 25287 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Apr 29 14:10:36 dist-test-slave-8n77 krb5kdc[25192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:10:36.095561 25298 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:36.085662 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:39227 (local address 127.2.197.254:36665)
0429 14:10:36.085827 (+ 165us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:36.085840 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:36.086125 (+ 285us) server_negotiation.cc:408] Connection header received
0429 14:10:36.086265 (+ 140us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:36.086301 (+ 36us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:36.086346 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:36.087416 (+ 1070us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:36.087531 (+ 115us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:36.088654 (+ 1123us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:36.090427 (+ 1773us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:36.090682 (+ 255us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:36.091918 (+ 1236us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:36.091924 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:36.091924 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:36.091936 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:36.093942 (+ 2006us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:36.094425 (+ 483us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:36.094428 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:36.094428 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:36.094501 (+ 73us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:36.094701 (+ 200us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:36.094702 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:36.094704 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:36.094985 (+ 281us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:36.095133 (+ 148us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:36.095258 (+ 125us) server_negotiation.cc:300] Negotiation successful
0429 14:10:36.095364 (+ 106us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":84,"threads_started":1}
I20260429 14:10:36.096575 2839 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260429 14:10:36.109607 25287 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:10:36.110354 25287 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:10:36.110736 25298 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:36.097980 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49967 (local address 127.2.197.254:36665)
0429 14:10:36.098020 (+ 40us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:36.098022 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:36.103836 (+ 5814us) server_negotiation.cc:408] Connection header received
0429 14:10:36.103900 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:36.103901 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:36.103912 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:36.103973 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:36.104820 (+ 847us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:36.105520 (+ 700us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:36.106257 (+ 737us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:36.106454 (+ 197us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:36.107646 (+ 1192us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:36.107650 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:36.107650 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:36.107657 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:36.109290 (+ 1633us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:36.109713 (+ 423us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:36.109714 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:36.109715 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:36.109740 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:36.109972 (+ 232us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:36.109974 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:36.109974 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:36.110083 (+ 109us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:36.110193 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:36.110392 (+ 199us) server_negotiation.cc:300] Negotiation successful
0429 14:10:36.110643 (+ 251us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260429 14:10:36.113992 25287 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 606bddb39aeb4393b0363d81c7ac591a: Generated new TSK 0
I20260429 14:10:36.114145 25287 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 14:10:36.119994 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 25208
W20260429 14:10:36.124424 25190 connection.cc:570] client connection to 127.2.197.254:36665 recv error: Network error: failed to read from TLS socket (remote: 127.2.197.254:36665): Connection reset by peer (error 104)
2026-04-29T14:10:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2335 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:10:36 dist-test-slave-8n77 krb5kdc[25307](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:10:36 dist-test-slave-8n77 krb5kdc[25307](info): set up 2 sockets
Apr 29 14:10:36 dist-test-slave-8n77 krb5kdc[25307](info): commencing operation
krb5kdc: starting...
W20260429 14:10:38.155807 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:10:38 dist-test-slave-8n77 krb5kdc[25307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:10:38Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:10:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:10:38.302675 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:42853
--webserver_interface=127.2.197.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:35165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:42853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:10:38.379637 25323 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:38.379846 25323 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:38.379885 25323 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:10:38.381439 25323 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:10:38.381520 25323 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:38.381543 25323 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:38.381574 25323 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:10:38.381598 25323 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:10:38.383329 25323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:35165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:42853
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:42853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:38.383603 25323 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:38.383790 25323 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260429 14:10:38.386518 25323 server_base.cc:1061] running on GCE node
W20260429 14:10:38.386498 25328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:38.386499 25331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:38.386498 25329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:38.386951 25323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:38.387210 25323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:38.388406 25323 hybrid_clock.cc:648] HybridClock initialized: now 1777471838388370 us; error 48 us; skew 500 ppm
Apr 29 14:10:38 dist-test-slave-8n77 krb5kdc[25307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471838, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:10:38.390842 25323 init.cc:377] Logged in from keytab as kudu/127.2.197.254@KRBTEST.COM (short username kudu)
I20260429 14:10:38.391407 25323 webserver.cc:492] Webserver started at http://127.2.197.254:36091/ using document root <none> and password file <none>
I20260429 14:10:38.391567 25323 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:38.391603 25323 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:38.391674 25323 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:38.392551 25323 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "f3780d58a62a455caf42064562c04872"
format_stamp: "Formatted at 2026-04-29 14:10:38 on dist-test-slave-8n77"
server_key: "cac0598402a64282633dc1da9a551962"
server_key_iv: "be43cee84e66f4e43ea5e81181e31e6e"
server_key_version: "encryptionkey@0"
I20260429 14:10:38.392813 25323 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "f3780d58a62a455caf42064562c04872"
format_stamp: "Formatted at 2026-04-29 14:10:38 on dist-test-slave-8n77"
server_key: "cac0598402a64282633dc1da9a551962"
server_key_iv: "be43cee84e66f4e43ea5e81181e31e6e"
server_key_version: "encryptionkey@0"
I20260429 14:10:38.394040 25323 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260429 14:10:38.394883 25338 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:38.395107 25323 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260429 14:10:38.395176 25323 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "f3780d58a62a455caf42064562c04872"
format_stamp: "Formatted at 2026-04-29 14:10:38 on dist-test-slave-8n77"
server_key: "cac0598402a64282633dc1da9a551962"
server_key_iv: "be43cee84e66f4e43ea5e81181e31e6e"
server_key_version: "encryptionkey@0"
I20260429 14:10:38.395229 25323 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:38.419777 25323 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:38.421663 25323 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:38.421775 25323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:38.425765 25323 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:42853
I20260429 14:10:38.425797 25390 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:42853 every 8 connection(s)
I20260429 14:10:38.426112 25323 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:10:38.426707 25391 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:38.427420 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 25323
I20260429 14:10:38.427675 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:10:38.427897 2839 external_mini_cluster.cc:1468] Setting key e0ea73ae288c68a84917ebf0b07f3348
I20260429 14:10:38.429559 25391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3780d58a62a455caf42064562c04872: Bootstrap starting.
I20260429 14:10:38.430123 25391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3780d58a62a455caf42064562c04872: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:38.430444 25391 log.cc:826] T 00000000000000000000000000000000 P f3780d58a62a455caf42064562c04872: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:38.431108 25391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3780d58a62a455caf42064562c04872: No bootstrap required, opened a new log
I20260429 14:10:38.432392 25391 raft_consensus.cc:359] T 00000000000000000000000000000000 P f3780d58a62a455caf42064562c04872 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3780d58a62a455caf42064562c04872" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 42853 } }
I20260429 14:10:38.432531 25394 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:38.428451 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33623 (local address 127.2.197.254:42853)
0429 14:10:38.428635 (+ 184us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:38.428644 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:38.428662 (+ 18us) server_negotiation.cc:408] Connection header received
0429 14:10:38.428976 (+ 314us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:38.428985 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:38.429023 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:38.429153 (+ 130us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:38.429536 (+ 383us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:38.430392 (+ 856us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:38.431047 (+ 655us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:38.431320 (+ 273us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:38.431843 (+ 523us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:38.431851 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:38.431851 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:10:38.432023 (+ 172us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:38.432042 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:10:38.432048 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:38.432150 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:38.432330 (+ 180us) server_negotiation.cc:300] Negotiation successful
0429 14:10:38.432356 (+ 26us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":78,"threads_started":1}
I20260429 14:10:38.432597 25391 raft_consensus.cc:385] T 00000000000000000000000000000000 P f3780d58a62a455caf42064562c04872 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:10:38.432705 25391 raft_consensus.cc:740] T 00000000000000000000000000000000 P f3780d58a62a455caf42064562c04872 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3780d58a62a455caf42064562c04872, State: Initialized, Role: FOLLOWER
I20260429 14:10:38.432811 25391 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3780d58a62a455caf42064562c04872 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3780d58a62a455caf42064562c04872" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 42853 } }
I20260429 14:10:38.432888 25391 raft_consensus.cc:399] T 00000000000000000000000000000000 P f3780d58a62a455caf42064562c04872 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:10:38.432916 25391 raft_consensus.cc:493] T 00000000000000000000000000000000 P f3780d58a62a455caf42064562c04872 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:10:38.432962 25391 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f3780d58a62a455caf42064562c04872 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:10:38.433662 2839 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260429 14:10:38.433710 25391 raft_consensus.cc:515] T 00000000000000000000000000000000 P f3780d58a62a455caf42064562c04872 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3780d58a62a455caf42064562c04872" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 42853 } }
I20260429 14:10:38.433835 25391 leader_election.cc:304] T 00000000000000000000000000000000 P f3780d58a62a455caf42064562c04872 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3780d58a62a455caf42064562c04872; no voters:
I20260429 14:10:38.434012 25391 leader_election.cc:290] T 00000000000000000000000000000000 P f3780d58a62a455caf42064562c04872 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:10:38.434044 25396 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f3780d58a62a455caf42064562c04872 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:10:38.434150 25396 raft_consensus.cc:697] T 00000000000000000000000000000000 P f3780d58a62a455caf42064562c04872 [term 1 LEADER]: Becoming Leader. State: Replica: f3780d58a62a455caf42064562c04872, State: Running, Role: LEADER
I20260429 14:10:38.434258 25391 sys_catalog.cc:565] T 00000000000000000000000000000000 P f3780d58a62a455caf42064562c04872 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:10:38.434314 25396 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3780d58a62a455caf42064562c04872 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3780d58a62a455caf42064562c04872" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 42853 } }
I20260429 14:10:38.434628 25396 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3780d58a62a455caf42064562c04872 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3780d58a62a455caf42064562c04872. Latest consensus state: current_term: 1 leader_uuid: "f3780d58a62a455caf42064562c04872" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3780d58a62a455caf42064562c04872" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 42853 } } }
I20260429 14:10:38.434634 25397 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3780d58a62a455caf42064562c04872 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3780d58a62a455caf42064562c04872" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3780d58a62a455caf42064562c04872" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 42853 } } }
I20260429 14:10:38.434718 25396 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3780d58a62a455caf42064562c04872 [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:38.434795 25397 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3780d58a62a455caf42064562c04872 [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:38.434962 25403 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:10:38.435566 25403 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:10:38.438463 25403 catalog_manager.cc:1357] Generated new cluster ID: 0bfbb45a433f4ac4894ae0bbe7204bd5
I20260429 14:10:38.438683 25403 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:10:38.438796 25394 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:38.435277 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57379 (local address 127.2.197.254:42853)
0429 14:10:38.435316 (+ 39us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:38.435317 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:38.435431 (+ 114us) server_negotiation.cc:408] Connection header received
0429 14:10:38.435535 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:38.435536 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:38.435547 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:38.435623 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:38.436016 (+ 393us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:38.436556 (+ 540us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:38.437433 (+ 877us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:38.437551 (+ 118us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:38.438341 (+ 790us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:38.438344 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:38.438344 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:10:38.438349 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:38.438364 (+ 15us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:10:38.438368 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:38.438407 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:38.438647 (+ 240us) server_negotiation.cc:300] Negotiation successful
0429 14:10:38.438660 (+ 13us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260429 14:10:38.445478 25403 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:10:38.446028 25403 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:10:38.451472 25403 catalog_manager.cc:6044] T 00000000000000000000000000000000 P f3780d58a62a455caf42064562c04872: Generated new TSK 0
I20260429 14:10:38.451669 25403 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 14:10:38.461130 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 25323
W20260429 14:10:38.465561 25305 connection.cc:570] client connection to 127.2.197.254:42853 recv error: Network error: failed to read from TLS socket (remote: 127.2.197.254:42853): Connection reset by peer (error 104)
2026-04-29T14:10:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2341 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:10:38 dist-test-slave-8n77 krb5kdc[25422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:10:38 dist-test-slave-8n77 krb5kdc[25422](info): set up 2 sockets
Apr 29 14:10:38 dist-test-slave-8n77 krb5kdc[25422](info): commencing operation
krb5kdc: starting...
W20260429 14:10:40.491614 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:10:40 dist-test-slave-8n77 krb5kdc[25422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:10:40Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:10:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:10:40.638500 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:34315
--webserver_interface=127.2.197.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:36845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:34315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:10:40.727746 25438 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:40.728024 25438 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:40.728063 25438 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:10:40.729674 25438 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:10:40.729770 25438 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:40.729794 25438 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:40.729831 25438 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:10:40.729861 25438 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:10:40.731690 25438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:36845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:34315
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:34315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:40.732194 25438 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:40.732405 25438 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:10:40.735143 25446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:40.735152 25444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:40.735152 25443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:40.735831 25438 server_base.cc:1061] running on GCE node
I20260429 14:10:40.736050 25438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:40.736322 25438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:40.737488 25438 hybrid_clock.cc:648] HybridClock initialized: now 1777471840737486 us; error 69 us; skew 500 ppm
Apr 29 14:10:40 dist-test-slave-8n77 krb5kdc[25422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471840, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:10:40.740154 25438 init.cc:377] Logged in from keytab as kudu/127.2.197.254@KRBTEST.COM (short username kudu)
I20260429 14:10:40.740784 25438 webserver.cc:492] Webserver started at http://127.2.197.254:40717/ using document root <none> and password file <none>
I20260429 14:10:40.740993 25438 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:40.741036 25438 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:40.741140 25438 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:40.742493 25438 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "1583da5d8caa43af9e0bb4374a84dab2"
format_stamp: "Formatted at 2026-04-29 14:10:40 on dist-test-slave-8n77"
server_key: "9e90bea80cc819cad2b09119bc76e569"
server_key_iv: "c12e791cd6ea97c7f1866c90d51144df"
server_key_version: "encryptionkey@0"
I20260429 14:10:40.742895 25438 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "1583da5d8caa43af9e0bb4374a84dab2"
format_stamp: "Formatted at 2026-04-29 14:10:40 on dist-test-slave-8n77"
server_key: "9e90bea80cc819cad2b09119bc76e569"
server_key_iv: "c12e791cd6ea97c7f1866c90d51144df"
server_key_version: "encryptionkey@0"
I20260429 14:10:40.744521 25438 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260429 14:10:40.745787 25453 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:40.746097 25438 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260429 14:10:40.746178 25438 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "1583da5d8caa43af9e0bb4374a84dab2"
format_stamp: "Formatted at 2026-04-29 14:10:40 on dist-test-slave-8n77"
server_key: "9e90bea80cc819cad2b09119bc76e569"
server_key_iv: "c12e791cd6ea97c7f1866c90d51144df"
server_key_version: "encryptionkey@0"
I20260429 14:10:40.746232 25438 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:40.749840 25438 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:40.752664 25438 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:40.752954 25438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:40.757332 25438 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:34315
I20260429 14:10:40.757613 25505 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:34315 every 8 connection(s)
I20260429 14:10:40.757807 25438 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:10:40.758500 25506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:40.760941 25506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1583da5d8caa43af9e0bb4374a84dab2: Bootstrap starting.
I20260429 14:10:40.761565 25506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1583da5d8caa43af9e0bb4374a84dab2: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:40.761832 25506 log.cc:826] T 00000000000000000000000000000000 P 1583da5d8caa43af9e0bb4374a84dab2: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:40.762792 25506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1583da5d8caa43af9e0bb4374a84dab2: No bootstrap required, opened a new log
I20260429 14:10:40.764026 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 25438
I20260429 14:10:40.764163 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:10:40.764340 2839 external_mini_cluster.cc:1468] Setting key b4ba948226e233e0f89abb33965ccf43
I20260429 14:10:40.764540 25506 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1583da5d8caa43af9e0bb4374a84dab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1583da5d8caa43af9e0bb4374a84dab2" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 34315 } }
I20260429 14:10:40.764712 25506 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1583da5d8caa43af9e0bb4374a84dab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:10:40.764750 25506 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1583da5d8caa43af9e0bb4374a84dab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1583da5d8caa43af9e0bb4374a84dab2, State: Initialized, Role: FOLLOWER
I20260429 14:10:40.764972 25506 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1583da5d8caa43af9e0bb4374a84dab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1583da5d8caa43af9e0bb4374a84dab2" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 34315 } }
I20260429 14:10:40.765079 25506 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1583da5d8caa43af9e0bb4374a84dab2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:10:40.765122 25506 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1583da5d8caa43af9e0bb4374a84dab2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:10:40.765167 25506 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1583da5d8caa43af9e0bb4374a84dab2 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:10:40.765870 25506 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1583da5d8caa43af9e0bb4374a84dab2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1583da5d8caa43af9e0bb4374a84dab2" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 34315 } }
I20260429 14:10:40.765998 25506 leader_election.cc:304] T 00000000000000000000000000000000 P 1583da5d8caa43af9e0bb4374a84dab2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1583da5d8caa43af9e0bb4374a84dab2; no voters:
I20260429 14:10:40.766211 25506 leader_election.cc:290] T 00000000000000000000000000000000 P 1583da5d8caa43af9e0bb4374a84dab2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:10:40.766300 25511 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1583da5d8caa43af9e0bb4374a84dab2 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:10:40.766530 25506 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1583da5d8caa43af9e0bb4374a84dab2 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:10:40.766407 25511 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1583da5d8caa43af9e0bb4374a84dab2 [term 1 LEADER]: Becoming Leader. State: Replica: 1583da5d8caa43af9e0bb4374a84dab2, State: Running, Role: LEADER
I20260429 14:10:40.766794 25511 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1583da5d8caa43af9e0bb4374a84dab2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1583da5d8caa43af9e0bb4374a84dab2" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 34315 } }
I20260429 14:10:40.767189 25510 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:40.765236 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:43803 (local address 127.2.197.254:34315)
0429 14:10:40.765481 (+ 245us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:40.765496 (+ 15us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:40.765521 (+ 25us) server_negotiation.cc:408] Connection header received
0429 14:10:40.765616 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:40.765622 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:40.765645 (+ 23us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:40.765761 (+ 116us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:40.766623 (+ 862us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:40.766627 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:40.766627 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:10:40.766633 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:40.766657 (+ 24us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:10:40.766665 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:40.766735 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:40.766894 (+ 159us) server_negotiation.cc:300] Negotiation successful
0429 14:10:40.767029 (+ 135us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":153,"threads_started":1}
I20260429 14:10:40.767823 25513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1583da5d8caa43af9e0bb4374a84dab2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1583da5d8caa43af9e0bb4374a84dab2. Latest consensus state: current_term: 1 leader_uuid: "1583da5d8caa43af9e0bb4374a84dab2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1583da5d8caa43af9e0bb4374a84dab2" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 34315 } } }
I20260429 14:10:40.767913 25513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1583da5d8caa43af9e0bb4374a84dab2 [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:40.767302 25512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1583da5d8caa43af9e0bb4374a84dab2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1583da5d8caa43af9e0bb4374a84dab2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1583da5d8caa43af9e0bb4374a84dab2" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 34315 } } }
I20260429 14:10:40.768065 25512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1583da5d8caa43af9e0bb4374a84dab2 [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:40.768298 2839 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
W20260429 14:10:40.768754 25527 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 1583da5d8caa43af9e0bb4374a84dab2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260429 14:10:40.768801 25527 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260429 14:10:40.768878 25524 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:10:40.769049 25524 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:10:40.770618 25524 catalog_manager.cc:1357] Generated new cluster ID: 0a75333f91e548aca93a220cb7c077c8
I20260429 14:10:40.770705 25524 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:10:40.771175 25510 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:40.769913 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:43187 (local address 127.2.197.254:34315)
0429 14:10:40.770008 (+ 95us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:40.770010 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:40.770142 (+ 132us) server_negotiation.cc:408] Connection header received
0429 14:10:40.770219 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:40.770220 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:40.770225 (+ 5us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:40.770312 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:40.770953 (+ 641us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:40.770955 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:40.770956 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:10:40.770960 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:40.770975 (+ 15us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:10:40.770979 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:40.771029 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:40.771104 (+ 75us) server_negotiation.cc:300] Negotiation successful
0429 14:10:40.771116 (+ 12us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260429 14:10:40.780750 25524 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:10:40.780942 25524 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:10:40.787796 25524 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 1583da5d8caa43af9e0bb4374a84dab2: Generated new TSK 0
I20260429 14:10:40.788005 25524 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 14:10:40.794431 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 25438
2026-04-29T14:10:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:10:40 dist-test-slave-8n77 krb5kdc[25537](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:10:40 dist-test-slave-8n77 krb5kdc[25537](info): set up 2 sockets
Apr 29 14:10:40 dist-test-slave-8n77 krb5kdc[25537](info): commencing operation
krb5kdc: starting...
W20260429 14:10:42.816036 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.002s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:10:42 dist-test-slave-8n77 krb5kdc[25537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:10:42Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:10:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:10:42.967059 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:46321
--webserver_interface=127.2.197.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:39105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:46321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:10:43.047700 25553 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:43.047974 25553 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:43.048014 25553 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:10:43.049685 25553 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:10:43.049774 25553 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:43.049799 25553 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:43.049820 25553 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:10:43.049851 25553 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:10:43.051682 25553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:39105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:46321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:46321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:43.051977 25553 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:43.052179 25553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:10:43.054818 25559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:43.054792 25558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:43.054776 25561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:43.055015 25553 server_base.cc:1061] running on GCE node
I20260429 14:10:43.055261 25553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:43.055464 25553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:43.056649 25553 hybrid_clock.cc:648] HybridClock initialized: now 1777471843056641 us; error 36 us; skew 500 ppm
Apr 29 14:10:43 dist-test-slave-8n77 krb5kdc[25537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471843, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:10:43.059284 25553 init.cc:377] Logged in from keytab as kudu/127.2.197.254@KRBTEST.COM (short username kudu)
I20260429 14:10:43.059886 25553 webserver.cc:492] Webserver started at http://127.2.197.254:45947/ using document root <none> and password file <none>
I20260429 14:10:43.060091 25553 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:43.060150 25553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:43.060289 25553 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:43.061223 25553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "a7f318264bb84e738ad96f562f00f165"
format_stamp: "Formatted at 2026-04-29 14:10:43 on dist-test-slave-8n77"
server_key: "59cfce8fec03561e168a009c6599c34d"
server_key_iv: "45502e87a7edbc6a3099c91979610c1f"
server_key_version: "encryptionkey@0"
I20260429 14:10:43.061558 25553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "a7f318264bb84e738ad96f562f00f165"
format_stamp: "Formatted at 2026-04-29 14:10:43 on dist-test-slave-8n77"
server_key: "59cfce8fec03561e168a009c6599c34d"
server_key_iv: "45502e87a7edbc6a3099c91979610c1f"
server_key_version: "encryptionkey@0"
I20260429 14:10:43.062835 25553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260429 14:10:43.063606 25568 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:43.063787 25553 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:10:43.063864 25553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "a7f318264bb84e738ad96f562f00f165"
format_stamp: "Formatted at 2026-04-29 14:10:43 on dist-test-slave-8n77"
server_key: "59cfce8fec03561e168a009c6599c34d"
server_key_iv: "45502e87a7edbc6a3099c91979610c1f"
server_key_version: "encryptionkey@0"
I20260429 14:10:43.063944 25553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:43.095492 25553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:43.102730 25553 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:43.102913 25553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:43.107064 25553 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:46321
I20260429 14:10:43.107092 25620 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:46321 every 8 connection(s)
I20260429 14:10:43.107419 25553 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:10:43.108021 25621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:43.110438 25621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7f318264bb84e738ad96f562f00f165: Bootstrap starting.
I20260429 14:10:43.111032 25621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7f318264bb84e738ad96f562f00f165: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:43.111287 25621 log.cc:826] T 00000000000000000000000000000000 P a7f318264bb84e738ad96f562f00f165: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:43.111931 25621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7f318264bb84e738ad96f562f00f165: No bootstrap required, opened a new log
I20260429 14:10:43.112362 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 25553
I20260429 14:10:43.112478 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:10:43.112648 2839 external_mini_cluster.cc:1468] Setting key 73e5e4a5c6297c343ca02ab64fb3e967
I20260429 14:10:43.113835 25621 raft_consensus.cc:359] T 00000000000000000000000000000000 P a7f318264bb84e738ad96f562f00f165 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f318264bb84e738ad96f562f00f165" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 46321 } }
I20260429 14:10:43.114022 25621 raft_consensus.cc:385] T 00000000000000000000000000000000 P a7f318264bb84e738ad96f562f00f165 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:10:43.114054 25621 raft_consensus.cc:740] T 00000000000000000000000000000000 P a7f318264bb84e738ad96f562f00f165 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7f318264bb84e738ad96f562f00f165, State: Initialized, Role: FOLLOWER
I20260429 14:10:43.114169 25621 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7f318264bb84e738ad96f562f00f165 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f318264bb84e738ad96f562f00f165" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 46321 } }
I20260429 14:10:43.114246 25621 raft_consensus.cc:399] T 00000000000000000000000000000000 P a7f318264bb84e738ad96f562f00f165 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:10:43.114310 25621 raft_consensus.cc:493] T 00000000000000000000000000000000 P a7f318264bb84e738ad96f562f00f165 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:10:43.114364 25621 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a7f318264bb84e738ad96f562f00f165 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:10:43.115118 25621 raft_consensus.cc:515] T 00000000000000000000000000000000 P a7f318264bb84e738ad96f562f00f165 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f318264bb84e738ad96f562f00f165" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 46321 } }
I20260429 14:10:43.115247 25621 leader_election.cc:304] T 00000000000000000000000000000000 P a7f318264bb84e738ad96f562f00f165 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7f318264bb84e738ad96f562f00f165; no voters:
I20260429 14:10:43.115411 25621 leader_election.cc:290] T 00000000000000000000000000000000 P a7f318264bb84e738ad96f562f00f165 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:10:43.115478 25626 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a7f318264bb84e738ad96f562f00f165 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:10:43.115612 25626 raft_consensus.cc:697] T 00000000000000000000000000000000 P a7f318264bb84e738ad96f562f00f165 [term 1 LEADER]: Becoming Leader. State: Replica: a7f318264bb84e738ad96f562f00f165, State: Running, Role: LEADER
I20260429 14:10:43.115658 25621 sys_catalog.cc:565] T 00000000000000000000000000000000 P a7f318264bb84e738ad96f562f00f165 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:10:43.117184 25626 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7f318264bb84e738ad96f562f00f165 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f318264bb84e738ad96f562f00f165" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 46321 } }
I20260429 14:10:43.118168 25628 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7f318264bb84e738ad96f562f00f165 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7f318264bb84e738ad96f562f00f165" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f318264bb84e738ad96f562f00f165" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 46321 } } }
I20260429 14:10:43.118302 25628 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7f318264bb84e738ad96f562f00f165 [sys.catalog]: This master's current role is: LEADER
Apr 29 14:10:43 dist-test-slave-8n77 krb5kdc[25537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:10:43.118518 25627 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7f318264bb84e738ad96f562f00f165 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7f318264bb84e738ad96f562f00f165. Latest consensus state: current_term: 1 leader_uuid: "a7f318264bb84e738ad96f562f00f165" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f318264bb84e738ad96f562f00f165" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 46321 } } }
I20260429 14:10:43.118690 25641 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:10:43.119289 25627 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7f318264bb84e738ad96f562f00f165 [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:43.119652 25641 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:10:43.122720 25641 catalog_manager.cc:1357] Generated new cluster ID: 8456b7c7fdf64b83a11fd973f089a536
I20260429 14:10:43.122777 25641 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:10:43.123027 25625 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:43.113162 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:39323 (local address 127.2.197.254:46321)
0429 14:10:43.113318 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:43.113327 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:43.113344 (+ 17us) server_negotiation.cc:408] Connection header received
0429 14:10:43.113734 (+ 390us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:43.113740 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:43.113777 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:43.113879 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:43.114965 (+ 1086us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:43.116056 (+ 1091us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:43.116814 (+ 758us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:43.117073 (+ 259us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:43.119595 (+ 2522us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:43.119604 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:43.119605 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:43.119619 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:43.121606 (+ 1987us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:43.121954 (+ 348us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:43.121957 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:43.121957 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:43.122019 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:43.122181 (+ 162us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:43.122182 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:43.122182 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:43.122448 (+ 266us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:43.122599 (+ 151us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:43.122746 (+ 147us) server_negotiation.cc:300] Negotiation successful
0429 14:10:43.122868 (+ 122us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":90,"threads_started":1}
I20260429 14:10:43.123867 2839 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260429 14:10:43.137836 25641 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:10:43.138545 25641 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:10:43.144541 25625 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:43.130642 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:56341 (local address 127.2.197.254:46321)
0429 14:10:43.130679 (+ 37us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:43.130681 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:43.137878 (+ 7197us) server_negotiation.cc:408] Connection header received
0429 14:10:43.137974 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:43.137976 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:43.138020 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:43.138097 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:43.138972 (+ 875us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:43.139679 (+ 707us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:43.140431 (+ 752us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:43.140908 (+ 477us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:43.141782 (+ 874us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:43.141786 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:43.141786 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:43.141792 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:43.143382 (+ 1590us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:43.143755 (+ 373us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:43.143756 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:43.143757 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:43.143779 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:43.143938 (+ 159us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:43.143939 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:43.143939 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:43.144033 (+ 94us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:43.144120 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:43.144335 (+ 215us) server_negotiation.cc:300] Negotiation successful
0429 14:10:43.144429 (+ 94us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260429 14:10:43.145035 25641 catalog_manager.cc:6044] T 00000000000000000000000000000000 P a7f318264bb84e738ad96f562f00f165: Generated new TSK 0
I20260429 14:10:43.145198 25641 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 14:10:43.152488 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 25553
2026-04-29T14:10:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2357 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:10:43 dist-test-slave-8n77 krb5kdc[25652](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:10:43 dist-test-slave-8n77 krb5kdc[25652](info): set up 2 sockets
Apr 29 14:10:43 dist-test-slave-8n77 krb5kdc[25652](info): commencing operation
krb5kdc: starting...
W20260429 14:10:45.183655 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:10:45 dist-test-slave-8n77 krb5kdc[25652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:10:45Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:10:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:10:45.326664 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:44829
--webserver_interface=127.2.197.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:45519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:44829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:10:45.407065 25668 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:45.407284 25668 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:45.407308 25668 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:10:45.408929 25668 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:10:45.408982 25668 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:45.408998 25668 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:45.409010 25668 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:10:45.409022 25668 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:10:45.410830 25668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:45519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:44829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:44829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:45.411078 25668 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:45.411321 25668 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:10:45.414054 25673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:45.414067 25676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:45.414053 25674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:45.414379 25668 server_base.cc:1061] running on GCE node
I20260429 14:10:45.414819 25668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:45.415084 25668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:45.416308 25668 hybrid_clock.cc:648] HybridClock initialized: now 1777471845416293 us; error 73 us; skew 500 ppm
Apr 29 14:10:45 dist-test-slave-8n77 krb5kdc[25652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471845, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:10:45.419019 25668 init.cc:377] Logged in from keytab as kudu/127.2.197.254@KRBTEST.COM (short username kudu)
I20260429 14:10:45.419658 25668 webserver.cc:492] Webserver started at http://127.2.197.254:43533/ using document root <none> and password file <none>
I20260429 14:10:45.419859 25668 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:45.419921 25668 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:45.420024 25668 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:45.420943 25668 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "abbd86df688940c99843b31f1f03f856"
format_stamp: "Formatted at 2026-04-29 14:10:45 on dist-test-slave-8n77"
server_key: "9bcecb4cbe3e4d213efef8bf152673e3"
server_key_iv: "2932c40e01f41411e177d81f9aa9c8f7"
server_key_version: "encryptionkey@0"
I20260429 14:10:45.421277 25668 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "abbd86df688940c99843b31f1f03f856"
format_stamp: "Formatted at 2026-04-29 14:10:45 on dist-test-slave-8n77"
server_key: "9bcecb4cbe3e4d213efef8bf152673e3"
server_key_iv: "2932c40e01f41411e177d81f9aa9c8f7"
server_key_version: "encryptionkey@0"
I20260429 14:10:45.422680 25668 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260429 14:10:45.423615 25683 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:45.423868 25668 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:10:45.423959 25668 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "abbd86df688940c99843b31f1f03f856"
format_stamp: "Formatted at 2026-04-29 14:10:45 on dist-test-slave-8n77"
server_key: "9bcecb4cbe3e4d213efef8bf152673e3"
server_key_iv: "2932c40e01f41411e177d81f9aa9c8f7"
server_key_version: "encryptionkey@0"
I20260429 14:10:45.424050 25668 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:45.446431 25668 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:45.448434 25668 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:45.448602 25668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:45.453759 25668 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:44829
I20260429 14:10:45.453784 25735 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:44829 every 8 connection(s)
I20260429 14:10:45.454169 25668 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:10:45.454775 25736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:45.457172 25736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abbd86df688940c99843b31f1f03f856: Bootstrap starting.
I20260429 14:10:45.457805 25736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abbd86df688940c99843b31f1f03f856: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:45.458073 25736 log.cc:826] T 00000000000000000000000000000000 P abbd86df688940c99843b31f1f03f856: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:45.458808 25736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abbd86df688940c99843b31f1f03f856: No bootstrap required, opened a new log
I20260429 14:10:45.460003 25736 raft_consensus.cc:359] T 00000000000000000000000000000000 P abbd86df688940c99843b31f1f03f856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abbd86df688940c99843b31f1f03f856" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 44829 } }
I20260429 14:10:45.460147 25736 raft_consensus.cc:385] T 00000000000000000000000000000000 P abbd86df688940c99843b31f1f03f856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:10:45.460187 25736 raft_consensus.cc:740] T 00000000000000000000000000000000 P abbd86df688940c99843b31f1f03f856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abbd86df688940c99843b31f1f03f856, State: Initialized, Role: FOLLOWER
I20260429 14:10:45.460340 25736 consensus_queue.cc:260] T 00000000000000000000000000000000 P abbd86df688940c99843b31f1f03f856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abbd86df688940c99843b31f1f03f856" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 44829 } }
I20260429 14:10:45.460414 25736 raft_consensus.cc:399] T 00000000000000000000000000000000 P abbd86df688940c99843b31f1f03f856 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:10:45.460456 25736 raft_consensus.cc:493] T 00000000000000000000000000000000 P abbd86df688940c99843b31f1f03f856 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:10:45.460505 25736 raft_consensus.cc:3060] T 00000000000000000000000000000000 P abbd86df688940c99843b31f1f03f856 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:10:45.461055 25736 raft_consensus.cc:515] T 00000000000000000000000000000000 P abbd86df688940c99843b31f1f03f856 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abbd86df688940c99843b31f1f03f856" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 44829 } }
I20260429 14:10:45.461196 25736 leader_election.cc:304] T 00000000000000000000000000000000 P abbd86df688940c99843b31f1f03f856 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abbd86df688940c99843b31f1f03f856; no voters:
I20260429 14:10:45.461378 25736 leader_election.cc:290] T 00000000000000000000000000000000 P abbd86df688940c99843b31f1f03f856 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:10:45.461445 25739 raft_consensus.cc:2804] T 00000000000000000000000000000000 P abbd86df688940c99843b31f1f03f856 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:10:45.461612 25736 sys_catalog.cc:565] T 00000000000000000000000000000000 P abbd86df688940c99843b31f1f03f856 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:10:45.461690 25739 raft_consensus.cc:697] T 00000000000000000000000000000000 P abbd86df688940c99843b31f1f03f856 [term 1 LEADER]: Becoming Leader. State: Replica: abbd86df688940c99843b31f1f03f856, State: Running, Role: LEADER
I20260429 14:10:45.461792 25739 consensus_queue.cc:237] T 00000000000000000000000000000000 P abbd86df688940c99843b31f1f03f856 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abbd86df688940c99843b31f1f03f856" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 44829 } }
I20260429 14:10:45.462004 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 25668
I20260429 14:10:45.462127 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:10:45.462364 2839 external_mini_cluster.cc:1468] Setting key b1e4e1669414670b14d4d2953f0c59c9
I20260429 14:10:45.462267 25740 sys_catalog.cc:455] T 00000000000000000000000000000000 P abbd86df688940c99843b31f1f03f856 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abbd86df688940c99843b31f1f03f856" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abbd86df688940c99843b31f1f03f856" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 44829 } } }
I20260429 14:10:45.462411 25740 sys_catalog.cc:458] T 00000000000000000000000000000000 P abbd86df688940c99843b31f1f03f856 [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:45.462670 25741 sys_catalog.cc:455] T 00000000000000000000000000000000 P abbd86df688940c99843b31f1f03f856 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abbd86df688940c99843b31f1f03f856. Latest consensus state: current_term: 1 leader_uuid: "abbd86df688940c99843b31f1f03f856" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abbd86df688940c99843b31f1f03f856" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 44829 } } }
I20260429 14:10:45.462782 25741 sys_catalog.cc:458] T 00000000000000000000000000000000 P abbd86df688940c99843b31f1f03f856 [sys.catalog]: This master's current role is: LEADER
W20260429 14:10:45.464000 25756 catalog_manager.cc:1568] T 00000000000000000000000000000000 P abbd86df688940c99843b31f1f03f856: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260429 14:10:45.464089 25756 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260429 14:10:45.464172 25748 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:10:45.464331 25748 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:10:45.465745 25748 catalog_manager.cc:1357] Generated new cluster ID: f0ee3127470846ca8286771fe2f62a52
I20260429 14:10:45.465801 25748 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:10:45.467442 25755 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:45.463063 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:45735 (local address 127.2.197.254:44829)
0429 14:10:45.463299 (+ 236us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:45.463309 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:45.463333 (+ 24us) server_negotiation.cc:408] Connection header received
0429 14:10:45.463497 (+ 164us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:45.463503 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:45.463549 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:45.463676 (+ 127us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:45.464057 (+ 381us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:45.464988 (+ 931us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:45.466208 (+ 1220us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:45.466498 (+ 290us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:45.466796 (+ 298us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:45.466800 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:45.466801 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:10:45.466983 (+ 182us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:45.467011 (+ 28us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:10:45.467017 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:45.467098 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:45.467202 (+ 104us) server_negotiation.cc:300] Negotiation successful
0429 14:10:45.467242 (+ 40us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
I20260429 14:10:45.468364 2839 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260429 14:10:45.477141 25748 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:10:45.477380 25755 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:45.470604 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60387 (local address 127.2.197.254:44829)
0429 14:10:45.470663 (+ 59us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:45.470665 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:45.474586 (+ 3921us) server_negotiation.cc:408] Connection header received
0429 14:10:45.474678 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:45.474679 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:45.474699 (+ 20us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:45.474765 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:45.475034 (+ 269us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:45.475732 (+ 698us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:45.476372 (+ 640us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:45.476563 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:45.477008 (+ 445us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:45.477012 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:45.477012 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:10:45.477017 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:45.477031 (+ 14us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:10:45.477035 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:45.477106 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:45.477190 (+ 84us) server_negotiation.cc:300] Negotiation successful
0429 14:10:45.477291 (+ 101us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260429 14:10:45.477950 25748 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:10:45.483669 25748 catalog_manager.cc:6044] T 00000000000000000000000000000000 P abbd86df688940c99843b31f1f03f856: Generated new TSK 0
I20260429 14:10:45.483834 25748 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 14:10:45.489042 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 25668
2026-04-29T14:10:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:10:45 dist-test-slave-8n77 krb5kdc[25767](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:10:45 dist-test-slave-8n77 krb5kdc[25767](info): set up 2 sockets
Apr 29 14:10:45 dist-test-slave-8n77 krb5kdc[25767](info): commencing operation
krb5kdc: starting...
W20260429 14:10:47.520314 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:10:47 dist-test-slave-8n77 krb5kdc[25767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:10:47Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:10:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:10:47.665835 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:42193
--webserver_interface=127.2.197.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:34873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:42193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:10:47.746631 25783 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:47.746872 25783 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:47.746914 25783 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:10:47.748467 25783 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:10:47.748549 25783 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:47.748572 25783 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:47.748592 25783 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:10:47.748623 25783 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:10:47.750267 25783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:34873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:42193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:42193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:47.750564 25783 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:47.750756 25783 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:10:47.753273 25789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:47.753329 25788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:47.753273 25783 server_base.cc:1061] running on GCE node
W20260429 14:10:47.753270 25791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:47.753705 25783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:47.753897 25783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:47.755033 25783 hybrid_clock.cc:648] HybridClock initialized: now 1777471847755020 us; error 30 us; skew 500 ppm
Apr 29 14:10:47 dist-test-slave-8n77 krb5kdc[25767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471847, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:10:47.757656 25783 init.cc:377] Logged in from keytab as kudu/127.2.197.254@KRBTEST.COM (short username kudu)
I20260429 14:10:47.758247 25783 webserver.cc:492] Webserver started at http://127.2.197.254:40735/ using document root <none> and password file <none>
I20260429 14:10:47.758497 25783 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:47.758565 25783 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:47.758682 25783 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:47.759665 25783 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "2d1e802d7da540e9a8c2daba018cd598"
format_stamp: "Formatted at 2026-04-29 14:10:47 on dist-test-slave-8n77"
server_key: "daab4a4d4de7eb84ed640c6111d3fc5f"
server_key_iv: "3c5878724014afaed14adf1645be4e5b"
server_key_version: "encryptionkey@0"
I20260429 14:10:47.760072 25783 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "2d1e802d7da540e9a8c2daba018cd598"
format_stamp: "Formatted at 2026-04-29 14:10:47 on dist-test-slave-8n77"
server_key: "daab4a4d4de7eb84ed640c6111d3fc5f"
server_key_iv: "3c5878724014afaed14adf1645be4e5b"
server_key_version: "encryptionkey@0"
I20260429 14:10:47.761579 25783 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260429 14:10:47.762502 25798 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:47.762701 25783 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:10:47.762792 25783 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "2d1e802d7da540e9a8c2daba018cd598"
format_stamp: "Formatted at 2026-04-29 14:10:47 on dist-test-slave-8n77"
server_key: "daab4a4d4de7eb84ed640c6111d3fc5f"
server_key_iv: "3c5878724014afaed14adf1645be4e5b"
server_key_version: "encryptionkey@0"
I20260429 14:10:47.762879 25783 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:47.766448 25783 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:47.768879 25783 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:47.769059 25783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:47.773205 25783 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:42193
I20260429 14:10:47.773411 25850 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:42193 every 8 connection(s)
I20260429 14:10:47.773629 25783 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:10:47.774246 25851 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:47.776610 25851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d1e802d7da540e9a8c2daba018cd598: Bootstrap starting.
I20260429 14:10:47.777158 25851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d1e802d7da540e9a8c2daba018cd598: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:47.777371 25851 log.cc:826] T 00000000000000000000000000000000 P 2d1e802d7da540e9a8c2daba018cd598: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:47.778016 25851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d1e802d7da540e9a8c2daba018cd598: No bootstrap required, opened a new log
I20260429 14:10:47.779243 25851 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2d1e802d7da540e9a8c2daba018cd598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d1e802d7da540e9a8c2daba018cd598" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 42193 } }
I20260429 14:10:47.779366 25851 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2d1e802d7da540e9a8c2daba018cd598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 14:10:47.779385 25851 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2d1e802d7da540e9a8c2daba018cd598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d1e802d7da540e9a8c2daba018cd598, State: Initialized, Role: FOLLOWER
I20260429 14:10:47.779474 25851 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d1e802d7da540e9a8c2daba018cd598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d1e802d7da540e9a8c2daba018cd598" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 42193 } }
I20260429 14:10:47.779511 25851 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2d1e802d7da540e9a8c2daba018cd598 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 14:10:47.779577 25851 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2d1e802d7da540e9a8c2daba018cd598 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 14:10:47.779613 25851 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2d1e802d7da540e9a8c2daba018cd598 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:10:47.780164 25851 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2d1e802d7da540e9a8c2daba018cd598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d1e802d7da540e9a8c2daba018cd598" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 42193 } }
I20260429 14:10:47.780303 25851 leader_election.cc:304] T 00000000000000000000000000000000 P 2d1e802d7da540e9a8c2daba018cd598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d1e802d7da540e9a8c2daba018cd598; no voters:
I20260429 14:10:47.780504 25851 leader_election.cc:290] T 00000000000000000000000000000000 P 2d1e802d7da540e9a8c2daba018cd598 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 14:10:47.780606 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 25783
I20260429 14:10:47.780591 25854 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2d1e802d7da540e9a8c2daba018cd598 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:10:47.780728 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:10:47.780747 25854 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2d1e802d7da540e9a8c2daba018cd598 [term 1 LEADER]: Becoming Leader. State: Replica: 2d1e802d7da540e9a8c2daba018cd598, State: Running, Role: LEADER
I20260429 14:10:47.780769 25851 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2d1e802d7da540e9a8c2daba018cd598 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:10:47.780864 25854 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d1e802d7da540e9a8c2daba018cd598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d1e802d7da540e9a8c2daba018cd598" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 42193 } }
I20260429 14:10:47.780920 2839 external_mini_cluster.cc:1468] Setting key f081606767cdc1aec74e264b3bf9d675
I20260429 14:10:47.781997 25855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d1e802d7da540e9a8c2daba018cd598 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d1e802d7da540e9a8c2daba018cd598" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d1e802d7da540e9a8c2daba018cd598" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 42193 } } }
I20260429 14:10:47.782100 25855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d1e802d7da540e9a8c2daba018cd598 [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:47.782437 25856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d1e802d7da540e9a8c2daba018cd598 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d1e802d7da540e9a8c2daba018cd598. Latest consensus state: current_term: 1 leader_uuid: "2d1e802d7da540e9a8c2daba018cd598" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d1e802d7da540e9a8c2daba018cd598" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 42193 } } }
I20260429 14:10:47.782506 25869 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:10:47.782505 25856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d1e802d7da540e9a8c2daba018cd598 [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:47.783414 25863 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:47.781572 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:38729 (local address 127.2.197.254:42193)
0429 14:10:47.781962 (+ 390us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:47.781976 (+ 14us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:47.782000 (+ 24us) server_negotiation.cc:408] Connection header received
0429 14:10:47.782111 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:47.782117 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:47.782147 (+ 30us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:47.782702 (+ 555us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:47.783037 (+ 335us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:47.783040 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:47.783040 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:10:47.783044 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:47.783063 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:10:47.783069 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:47.783154 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:47.783236 (+ 82us) server_negotiation.cc:300] Negotiation successful
0429 14:10:47.783270 (+ 34us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":80,"threads_started":1}
I20260429 14:10:47.783488 25869 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:10:47.784576 2839 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260429 14:10:47.785123 25869 catalog_manager.cc:1357] Generated new cluster ID: 04adfc74f0484dc6a83aeea748f9422e
I20260429 14:10:47.785182 25869 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:10:47.787427 25863 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:47.786146 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60639 (local address 127.2.197.254:42193)
0429 14:10:47.786427 (+ 281us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:47.786429 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:47.786440 (+ 11us) server_negotiation.cc:408] Connection header received
0429 14:10:47.786464 (+ 24us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:47.786465 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:47.786471 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:47.786543 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:47.787207 (+ 664us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:47.787208 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:47.787209 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 14:10:47.787213 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:47.787230 (+ 17us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 14:10:47.787234 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:47.787298 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:47.787351 (+ 53us) server_negotiation.cc:300] Negotiation successful
0429 14:10:47.787365 (+ 14us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257}
I20260429 14:10:47.818095 25869 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:10:47.818251 25869 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:10:47.821537 25869 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 2d1e802d7da540e9a8c2daba018cd598: Generated new TSK 0
I20260429 14:10:47.821717 25869 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 14:10:47.833603 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 25783
2026-04-29T14:10:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2344 ms)
[----------] 18 tests from AuthTokenIssuingTest (42037 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:10:47 dist-test-slave-8n77 krb5kdc[25882](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:10:47 dist-test-slave-8n77 krb5kdc[25882](info): set up 2 sockets
Apr 29 14:10:47 dist-test-slave-8n77 krb5kdc[25882](info): commencing operation
krb5kdc: starting...
W20260429 14:10:49.859961 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.007s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:10:49 dist-test-slave-8n77 krb5kdc[25882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:10:49Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:10:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:10:50.015038 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:33447
--webserver_interface=127.2.197.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:34823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:33447,127.2.197.253:44975,127.2.197.252:46339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:10:50.097618 25898 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260429 14:10:50.097887 25898 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:50.097919 25898 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:50.097963 25898 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:10:50.099597 25898 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:10:50.099690 25898 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:50.099714 25898 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:50.099735 25898 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:10:50.099768 25898 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:10:50.101490 25898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:34823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:33447,127.2.197.253:44975,127.2.197.252:46339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:33447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:50.101794 25898 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:50.101985 25898 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:10:50.104663 25904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:50.104708 25906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:50.104676 25903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:50.104930 25898 server_base.cc:1061] running on GCE node
I20260429 14:10:50.105160 25898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:50.105430 25898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:50.106573 25898 hybrid_clock.cc:648] HybridClock initialized: now 1777471850106549 us; error 33 us; skew 500 ppm
Apr 29 14:10:50 dist-test-slave-8n77 krb5kdc[25882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471850, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:10:50.109055 25898 init.cc:377] Logged in from keytab as kudu/127.2.197.254@KRBTEST.COM (short username kudu)
I20260429 14:10:50.109652 25898 webserver.cc:492] Webserver started at http://127.2.197.254:40779/ using document root <none> and password file <none>
I20260429 14:10:50.109856 25898 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:50.109925 25898 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:50.110030 25898 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:50.111042 25898 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "ff9d971dc5f54ba3a3c23958a8e1ef99"
format_stamp: "Formatted at 2026-04-29 14:10:50 on dist-test-slave-8n77"
server_key: "633fba75a00451c2b64791c037b91120"
server_key_iv: "1697fc1635f522cad08148854476a0ad"
server_key_version: "encryptionkey@0"
I20260429 14:10:50.111469 25898 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "ff9d971dc5f54ba3a3c23958a8e1ef99"
format_stamp: "Formatted at 2026-04-29 14:10:50 on dist-test-slave-8n77"
server_key: "633fba75a00451c2b64791c037b91120"
server_key_iv: "1697fc1635f522cad08148854476a0ad"
server_key_version: "encryptionkey@0"
I20260429 14:10:50.112716 25898 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260429 14:10:50.113507 25913 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:50.113689 25898 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:10:50.113772 25898 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "ff9d971dc5f54ba3a3c23958a8e1ef99"
format_stamp: "Formatted at 2026-04-29 14:10:50 on dist-test-slave-8n77"
server_key: "633fba75a00451c2b64791c037b91120"
server_key_iv: "1697fc1635f522cad08148854476a0ad"
server_key_version: "encryptionkey@0"
I20260429 14:10:50.113853 25898 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:50.123721 25898 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:50.130750 25898 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:50.130929 25898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:50.135281 25898 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:33447
I20260429 14:10:50.135326 25965 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:33447 every 8 connection(s)
I20260429 14:10:50.135656 25898 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:10:50.136225 25966 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:50.137933 25966 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.197.254" port: 33447 } has no permanent_uuid. Determining permanent_uuid...
I20260429 14:10:50.140502 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 25898
I20260429 14:10:50.140610 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:10:50.140810 2839 external_mini_cluster.cc:1468] Setting key 4915905f8a2e7be89c6dbbea1d933b0a
Apr 29 14:10:50 dist-test-slave-8n77 krb5kdc[25882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471850, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.197.253@KRBTEST.COM; defaulting to no policy
I20260429 14:10:50.148129 25968 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.138578 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:33447 (local address 127.0.0.1:60844)
0429 14:10:50.138707 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:50.138736 (+ 29us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:50.139066 (+ 330us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:50.139381 (+ 315us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:50.139383 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:50.139678 (+ 295us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:10:50.139922 (+ 244us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:50.139926 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.141259 (+ 1333us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:50.141260 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:50.141615 (+ 355us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:50.141617 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.141764 (+ 147us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:50.142474 (+ 710us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:10:50.142488 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:10:50.144165 (+ 1677us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:10:50.146718 (+ 2553us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:50.146724 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:50.146726 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:50.146910 (+ 184us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:50.147138 (+ 228us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:50.147139 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:50.147140 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:50.147195 (+ 55us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:50.147573 (+ 378us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:10:50.147577 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:10:50.147644 (+ 67us) client_negotiation.cc:770] Sending connection context
0429 14:10:50.147747 (+ 103us) client_negotiation.cc:241] Negotiation successful
0429 14:10:50.147980 (+ 233us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":108,"thread_start_us":60,"threads_started":1}
I20260429 14:10:50.148129 25969 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.138702 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60844 (local address 127.2.197.254:33447)
0429 14:10:50.138840 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:50.138846 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:50.138866 (+ 20us) server_negotiation.cc:408] Connection header received
0429 14:10:50.139210 (+ 344us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:50.139213 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:50.139238 (+ 25us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:50.139315 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:50.140148 (+ 833us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.141123 (+ 975us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:50.141704 (+ 581us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.141890 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:50.144278 (+ 2388us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:50.144287 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:50.144288 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:50.144301 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:50.146598 (+ 2297us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:50.147022 (+ 424us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:50.147025 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:50.147026 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:50.147070 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:50.147258 (+ 188us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:50.147259 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:50.147260 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:50.147455 (+ 195us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:50.147595 (+ 140us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:50.147773 (+ 178us) server_negotiation.cc:300] Negotiation successful
0429 14:10:50.147915 (+ 142us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
I20260429 14:10:50.148839 25966 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44975 } has no permanent_uuid. Determining permanent_uuid...
I20260429 14:10:50.149284 25968 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.149156 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.253:44975 (local address 127.0.0.1:32814)
0429 14:10:50.149216 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:50.149262 (+ 46us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.253:44975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260429 14:10:50.149415 25914 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.197.253:44975: connect: Connection refused (error 111)
I20260429 14:10:50.149940 25968 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.149762 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.253:44975 (local address 127.0.0.1:32816)
0429 14:10:50.149859 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:50.149889 (+ 30us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.253:44975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260429 14:10:50.150089 25966 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.197.253:44975: Network error: Client connection negotiation failed: client connection to 127.2.197.253:44975: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:10:50.190593 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.253:44975
--webserver_interface=127.2.197.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:34823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:33447,127.2.197.253:44975,127.2.197.252:46339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260429 14:10:50.199290 25966 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44975 } attempt: 1
I20260429 14:10:50.199818 25968 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.199690 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.253:44975 (local address 127.0.0.1:32832)
0429 14:10:50.199752 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:50.199790 (+ 38us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.253:44975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260429 14:10:50.200433 25968 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.200299 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.253:44975 (local address 127.0.0.1:32848)
0429 14:10:50.200368 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:50.200398 (+ 30us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.253:44975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260429 14:10:50.200569 25966 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.197.253:44975: Network error: Client connection negotiation failed: client connection to 127.2.197.253:44975: connect: Connection refused (error 111)
I20260429 14:10:50.268978 25966 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44975 } attempt: 2
I20260429 14:10:50.269922 25968 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.269670 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.253:44975 (local address 127.0.0.1:32850)
0429 14:10:50.269792 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:50.269860 (+ 68us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.253:44975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260429 14:10:50.270740 25968 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.270598 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.253:44975 (local address 127.0.0.1:32860)
0429 14:10:50.270668 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:50.270708 (+ 40us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.253:44975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260429 14:10:50.271103 25966 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.197.253:44975: Network error: Client connection negotiation failed: client connection to 127.2.197.253:44975: connect: Connection refused (error 111)
W20260429 14:10:50.274780 25975 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260429 14:10:50.274993 25975 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:50.275012 25975 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:50.275034 25975 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:10:50.276688 25975 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:10:50.276746 25975 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:50.276760 25975 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:50.276773 25975 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:10:50.276785 25975 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:10:50.278640 25975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:34823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:33447,127.2.197.253:44975,127.2.197.252:46339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.253:44975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.197.253
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:50.278891 25975 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:50.279093 25975 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:10:50.281792 25983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:50.281824 25981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:50.281816 25980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:50.281994 25975 server_base.cc:1061] running on GCE node
I20260429 14:10:50.282341 25975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:50.282578 25975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:50.283726 25975 hybrid_clock.cc:648] HybridClock initialized: now 1777471850283715 us; error 31 us; skew 500 ppm
Apr 29 14:10:50 dist-test-slave-8n77 krb5kdc[25882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471850, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:10:50.286248 25975 init.cc:377] Logged in from keytab as kudu/127.2.197.253@KRBTEST.COM (short username kudu)
I20260429 14:10:50.287379 25975 webserver.cc:492] Webserver started at http://127.2.197.253:34437/ using document root <none> and password file <none>
I20260429 14:10:50.287680 25975 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:50.287755 25975 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:50.287881 25975 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:50.289150 25975 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-1/data/instance:
uuid: "30459ec1650647c8a6158f0ef433d148"
format_stamp: "Formatted at 2026-04-29 14:10:50 on dist-test-slave-8n77"
server_key: "ef8af69aa13abc74a799b6bc4771c21a"
server_key_iv: "2cb307373c57eb284f15ebf2bd8aec08"
server_key_version: "encryptionkey@0"
I20260429 14:10:50.289541 25975 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-1/wal/instance:
uuid: "30459ec1650647c8a6158f0ef433d148"
format_stamp: "Formatted at 2026-04-29 14:10:50 on dist-test-slave-8n77"
server_key: "ef8af69aa13abc74a799b6bc4771c21a"
server_key_iv: "2cb307373c57eb284f15ebf2bd8aec08"
server_key_version: "encryptionkey@0"
I20260429 14:10:50.291083 25975 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260429 14:10:50.292024 25990 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:50.292300 25975 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:10:50.292383 25975 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-1/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-1/wal
uuid: "30459ec1650647c8a6158f0ef433d148"
format_stamp: "Formatted at 2026-04-29 14:10:50 on dist-test-slave-8n77"
server_key: "ef8af69aa13abc74a799b6bc4771c21a"
server_key_iv: "2cb307373c57eb284f15ebf2bd8aec08"
server_key_version: "encryptionkey@0"
I20260429 14:10:50.292461 25975 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:50.314581 25975 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:50.316673 25975 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:50.316851 25975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:50.321050 25975 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.253:44975
I20260429 14:10:50.321117 26042 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.253:44975 every 8 connection(s)
I20260429 14:10:50.321450 25975 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-1/data/info.pb
I20260429 14:10:50.322078 26043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:50.323853 26043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.197.254" port: 33447 } has no permanent_uuid. Determining permanent_uuid...
I20260429 14:10:50.326037 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 25975
I20260429 14:10:50.326165 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-1/wal/instance
I20260429 14:10:50.326370 2839 external_mini_cluster.cc:1468] Setting key c5a0dcb08b10965e8db39c966d5be830
Apr 29 14:10:50 dist-test-slave-8n77 krb5kdc[25882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471850, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.253@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:10:50.333129 26045 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.324669 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:33447 (local address 127.0.0.1:60852)
0429 14:10:50.324848 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:50.324877 (+ 29us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:50.325262 (+ 385us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:50.325512 (+ 250us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:50.325515 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:50.325811 (+ 296us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:10:50.326221 (+ 410us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:50.326226 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.327135 (+ 909us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:50.327137 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:50.327744 (+ 607us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:50.327747 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.327897 (+ 150us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:50.328565 (+ 668us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:10:50.328575 (+ 10us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:10:50.330025 (+ 1450us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:10:50.331962 (+ 1937us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:50.331968 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:50.331970 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:50.332164 (+ 194us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:50.332350 (+ 186us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:50.332350 (+ 0us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:50.332351 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:50.332404 (+ 53us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:50.332687 (+ 283us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:10:50.332691 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:10:50.332799 (+ 108us) client_negotiation.cc:770] Sending connection context
0429 14:10:50.332881 (+ 82us) client_negotiation.cc:241] Negotiation successful
0429 14:10:50.332984 (+ 103us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":98,"threads_started":1}
I20260429 14:10:50.333137 25969 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.324683 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60852 (local address 127.2.197.254:33447)
0429 14:10:50.324757 (+ 74us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:50.324759 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:50.324958 (+ 199us) server_negotiation.cc:408] Connection header received
0429 14:10:50.325364 (+ 406us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:50.325365 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:50.325379 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:50.325437 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:50.326404 (+ 967us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.326916 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:50.327858 (+ 942us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.328108 (+ 250us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:50.330144 (+ 2036us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:50.330149 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:50.330150 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:50.330159 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:50.331873 (+ 1714us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:50.332249 (+ 376us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:50.332251 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:50.332251 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:50.332281 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:50.332494 (+ 213us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:50.332495 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:50.332496 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:50.332623 (+ 127us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:50.332696 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:50.332927 (+ 231us) server_negotiation.cc:300] Negotiation successful
0429 14:10:50.333018 (+ 91us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260429 14:10:50.333837 26043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44975 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.197.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.252@KRBTEST.COM" created.
Apr 29 14:10:50 dist-test-slave-8n77 krb5kdc[25882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471850, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.253@KRBTEST.COM for kudu/127.2.197.253@KRBTEST.COM
I20260429 14:10:50.341773 26045 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.334116 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.253:44975 (local address 127.0.0.1:32864)
0429 14:10:50.334177 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:50.334190 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:50.334239 (+ 49us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:50.334488 (+ 249us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:50.334489 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:50.334661 (+ 172us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:10:50.334862 (+ 201us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:50.334864 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.335740 (+ 876us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:50.335741 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:50.336081 (+ 340us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:50.336082 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.336222 (+ 140us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:50.336613 (+ 391us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:10:50.336618 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:10:50.338909 (+ 2291us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:10:50.340854 (+ 1945us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:50.340856 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:50.340856 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:50.340992 (+ 136us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:50.341173 (+ 181us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:50.341174 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:50.341175 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:50.341201 (+ 26us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:50.341433 (+ 232us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:10:50.341434 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:10:50.341478 (+ 44us) client_negotiation.cc:770] Sending connection context
0429 14:10:50.341548 (+ 70us) client_negotiation.cc:241] Negotiation successful
0429 14:10:50.341658 (+ 110us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260429 14:10:50.341842 26047 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.334153 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:32864 (local address 127.2.197.253:44975)
0429 14:10:50.334337 (+ 184us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:50.334343 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:50.334359 (+ 16us) server_negotiation.cc:408] Connection header received
0429 14:10:50.334380 (+ 21us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:50.334387 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:50.334398 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:50.334455 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:50.334941 (+ 486us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.335662 (+ 721us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:50.336230 (+ 568us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.336688 (+ 458us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:50.338997 (+ 2309us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:50.339007 (+ 10us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:50.339008 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:50.339021 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:50.340756 (+ 1735us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:50.341079 (+ 323us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:50.341081 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:50.341082 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:50.341112 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:50.341255 (+ 143us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:50.341256 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:50.341257 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:50.341370 (+ 113us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:50.341456 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:50.341578 (+ 122us) server_negotiation.cc:300] Negotiation successful
0429 14:10:50.341723 (+ 145us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":158,"thread_start_us":105,"threads_started":1}
I20260429 14:10:50.342419 26043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.197.252" port: 46339 } has no permanent_uuid. Determining permanent_uuid...
I20260429 14:10:50.342851 26045 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.342699 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:46339 (local address 127.0.0.1:53280)
0429 14:10:50.342763 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:50.342821 (+ 58us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.252:46339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260429 14:10:50.342986 25993 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.197.252:46339: connect: Connection refused (error 111)
I20260429 14:10:50.343410 26045 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.343304 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:46339 (local address 127.0.0.1:53296)
0429 14:10:50.343356 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:50.343386 (+ 30us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.252:46339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260429 14:10:50.343537 26043 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.197.252:46339: Network error: Client connection negotiation failed: client connection to 127.2.197.252:46339: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.252@KRBTEST.COM" created.
I20260429 14:10:50.362352 25966 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44975 } attempt: 3
Apr 29 14:10:50 dist-test-slave-8n77 krb5kdc[25882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471850, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for kudu/127.2.197.253@KRBTEST.COM
I20260429 14:10:50.369552 25968 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.362752 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.253:44975 (local address 127.0.0.1:32878)
0429 14:10:50.362828 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:50.362844 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:50.362912 (+ 68us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:50.363078 (+ 166us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:50.363079 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:50.363273 (+ 194us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:10:50.363539 (+ 266us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:50.363542 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.364166 (+ 624us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:50.364167 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:50.364735 (+ 568us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:50.364738 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.364818 (+ 80us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:50.365293 (+ 475us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:10:50.365299 (+ 6us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:10:50.366758 (+ 1459us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:10:50.368628 (+ 1870us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:50.368630 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:50.368630 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:50.368774 (+ 144us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:50.368960 (+ 186us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:50.368961 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:50.368961 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:50.368985 (+ 24us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:50.369279 (+ 294us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:10:50.369280 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:10:50.369335 (+ 55us) client_negotiation.cc:770] Sending connection context
0429 14:10:50.369396 (+ 61us) client_negotiation.cc:241] Negotiation successful
0429 14:10:50.369479 (+ 83us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260429 14:10:50.369660 26047 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.362844 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:32878 (local address 127.2.197.253:44975)
0429 14:10:50.362906 (+ 62us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:50.362908 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:50.362919 (+ 11us) server_negotiation.cc:408] Connection header received
0429 14:10:50.362987 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:50.362988 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:50.362999 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:50.363049 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:50.363624 (+ 575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.364076 (+ 452us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:50.364826 (+ 750us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.364983 (+ 157us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:50.366849 (+ 1866us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:50.366853 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:50.366854 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:50.366863 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:50.368519 (+ 1656us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:50.368859 (+ 340us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:50.368860 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:50.368861 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:50.368890 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:50.369049 (+ 159us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:50.369050 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:50.369050 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:50.369181 (+ 131us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:50.369257 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:50.369450 (+ 193us) server_negotiation.cc:300] Negotiation successful
0429 14:10:50.369540 (+ 90us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260429 14:10:50.370213 25966 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.197.252" port: 46339 } has no permanent_uuid. Determining permanent_uuid...
I20260429 14:10:50.370838 25968 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.370622 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:46339 (local address 127.0.0.1:53298)
0429 14:10:50.370701 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:50.370734 (+ 33us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.252:46339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260429 14:10:50.371558 25968 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.371427 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:46339 (local address 127.0.0.1:53308)
0429 14:10:50.371511 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:50.371539 (+ 28us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.252:46339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260429 14:10:50.371984 25966 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.197.252:46339: Network error: Client connection negotiation failed: client connection to 127.2.197.252:46339: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:10:50.382853 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.252:46339
--webserver_interface=127.2.197.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:34823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:33447,127.2.197.253:44975,127.2.197.252:46339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260429 14:10:50.392772 26043 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.197.252" port: 46339 } attempt: 1
I20260429 14:10:50.393402 26045 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.393239 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:46339 (local address 127.0.0.1:53312)
0429 14:10:50.393330 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:50.393371 (+ 41us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.252:46339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260429 14:10:50.393940 26045 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.393748 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:46339 (local address 127.0.0.1:53326)
0429 14:10:50.393797 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:50.393915 (+ 118us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.252:46339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260429 14:10:50.394090 26043 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.197.252:46339: Network error: Client connection negotiation failed: client connection to 127.2.197.252:46339: connect: Connection refused (error 111)
I20260429 14:10:50.403223 25966 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.197.252" port: 46339 } attempt: 1
I20260429 14:10:50.403738 25968 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.403572 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:46339 (local address 127.0.0.1:53336)
0429 14:10:50.403644 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:50.403697 (+ 53us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.252:46339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260429 14:10:50.404186 25968 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.404093 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:46339 (local address 127.0.0.1:53338)
0429 14:10:50.404139 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:50.404167 (+ 28us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.252:46339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260429 14:10:50.404337 25966 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.197.252:46339: Network error: Client connection negotiation failed: client connection to 127.2.197.252:46339: connect: Connection refused (error 111)
I20260429 14:10:50.462335 26043 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.197.252" port: 46339 } attempt: 2
I20260429 14:10:50.462903 26045 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.462765 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:46339 (local address 127.0.0.1:53354)
0429 14:10:50.462833 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:50.462875 (+ 42us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.252:46339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260429 14:10:50.462968 26052 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260429 14:10:50.463274 26052 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:50.463339 26052 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:50.463368 26052 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260429 14:10:50.463424 26045 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.463212 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:46339 (local address 127.0.0.1:53364)
0429 14:10:50.463269 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:50.463300 (+ 31us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.252:46339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260429 14:10:50.463529 26043 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.197.252:46339: Network error: Client connection negotiation failed: client connection to 127.2.197.252:46339: connect: Connection refused (error 111)
W20260429 14:10:50.464977 26052 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:10:50.465023 26052 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:50.465037 26052 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:50.465050 26052 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:10:50.465062 26052 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:10:50.466828 26052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:34823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:33447,127.2.197.253:44975,127.2.197.252:46339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.252:46339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.197.252
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:50.467049 26052 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:50.467192 26052 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:10:50.469487 26058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:50.469484 26060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:50.469487 26057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:50.469617 26052 server_base.cc:1061] running on GCE node
I20260429 14:10:50.469923 26052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:50.470165 26052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:50.471335 26052 hybrid_clock.cc:648] HybridClock initialized: now 1777471850471330 us; error 40 us; skew 500 ppm
Apr 29 14:10:50 dist-test-slave-8n77 krb5kdc[25882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471850, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:10:50.473815 26052 init.cc:377] Logged in from keytab as kudu/127.2.197.252@KRBTEST.COM (short username kudu)
I20260429 14:10:50.474402 26052 webserver.cc:492] Webserver started at http://127.2.197.252:43753/ using document root <none> and password file <none>
I20260429 14:10:50.474601 26052 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:50.474661 26052 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:50.474771 26052 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:50.475699 26052 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-2/data/instance:
uuid: "b19fb2f308f4439da00136d57a251789"
format_stamp: "Formatted at 2026-04-29 14:10:50 on dist-test-slave-8n77"
server_key: "f60b0abc3456cf84c82747e49c44c678"
server_key_iv: "b1a8710ff3cc804fc2a82dc76619b25b"
server_key_version: "encryptionkey@0"
I20260429 14:10:50.476027 26052 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-2/wal/instance:
uuid: "b19fb2f308f4439da00136d57a251789"
format_stamp: "Formatted at 2026-04-29 14:10:50 on dist-test-slave-8n77"
server_key: "f60b0abc3456cf84c82747e49c44c678"
server_key_iv: "b1a8710ff3cc804fc2a82dc76619b25b"
server_key_version: "encryptionkey@0"
I20260429 14:10:50.477213 26052 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260429 14:10:50.477979 26067 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:50.478268 26052 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:10:50.478413 26052 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-2/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-2/wal
uuid: "b19fb2f308f4439da00136d57a251789"
format_stamp: "Formatted at 2026-04-29 14:10:50 on dist-test-slave-8n77"
server_key: "f60b0abc3456cf84c82747e49c44c678"
server_key_iv: "b1a8710ff3cc804fc2a82dc76619b25b"
server_key_version: "encryptionkey@0"
I20260429 14:10:50.478493 26052 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:50.479521 25966 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.197.252" port: 46339 } attempt: 2
I20260429 14:10:50.479960 25968 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.479810 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:46339 (local address 127.0.0.1:53372)
0429 14:10:50.479878 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:50.479919 (+ 41us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.252:46339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260429 14:10:50.480502 25968 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.480354 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:46339 (local address 127.0.0.1:53384)
0429 14:10:50.480437 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:50.480473 (+ 36us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.252:46339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260429 14:10:50.480643 25966 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.197.252:46339: Network error: Client connection negotiation failed: client connection to 127.2.197.252:46339: connect: Connection refused (error 111)
I20260429 14:10:50.493616 26052 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:50.495617 26052 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:50.495762 26052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:50.499925 26052 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.252:46339
I20260429 14:10:50.500002 26119 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.252:46339 every 8 connection(s)
I20260429 14:10:50.500298 26052 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-2/data/info.pb
I20260429 14:10:50.500849 26120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:50.502761 26120 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.197.254" port: 33447 } has no permanent_uuid. Determining permanent_uuid...
Apr 29 14:10:50 dist-test-slave-8n77 krb5kdc[25882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471850, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.252@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:10:50.507529 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 26052
I20260429 14:10:50.507651 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777471413505459-2839-0/minicluster-data/master-2/wal/instance
I20260429 14:10:50.507834 2839 external_mini_cluster.cc:1468] Setting key dc2120961e7ce5aee20d6dceb66eec52
I20260429 14:10:50.511049 25969 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.503528 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60860 (local address 127.2.197.254:33447)
0429 14:10:50.503617 (+ 89us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:50.503619 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:50.503754 (+ 135us) server_negotiation.cc:408] Connection header received
0429 14:10:50.504135 (+ 381us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:50.504136 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:50.504149 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:50.504246 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:50.504915 (+ 669us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.505347 (+ 432us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:50.505901 (+ 554us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.506077 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:50.508155 (+ 2078us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:50.508159 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:50.508160 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:50.508167 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:50.509806 (+ 1639us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:50.510226 (+ 420us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:50.510226 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:50.510227 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:50.510245 (+ 18us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:50.510478 (+ 233us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:50.510478 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:50.510479 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:50.510559 (+ 80us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:50.510611 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:50.510882 (+ 271us) server_negotiation.cc:300] Negotiation successful
0429 14:10:50.510961 (+ 79us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260429 14:10:50.511085 26122 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.503437 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:33447 (local address 127.0.0.1:60860)
0429 14:10:50.503643 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:50.503678 (+ 35us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:50.504040 (+ 362us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:50.504266 (+ 226us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:50.504270 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:50.504576 (+ 306us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:10:50.504823 (+ 247us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:50.504826 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.505448 (+ 622us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:50.505451 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:50.505815 (+ 364us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:50.505816 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.505945 (+ 129us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:50.506565 (+ 620us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:10:50.506574 (+ 9us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:10:50.508049 (+ 1475us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:10:50.509873 (+ 1824us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:50.509877 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:50.509879 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:50.510139 (+ 260us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:50.510341 (+ 202us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:50.510342 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:50.510342 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:50.510416 (+ 74us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:50.510622 (+ 206us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:10:50.510626 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:10:50.510764 (+ 138us) client_negotiation.cc:770] Sending connection context
0429 14:10:50.510846 (+ 82us) client_negotiation.cc:241] Negotiation successful
0429 14:10:50.510941 (+ 95us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":90,"threads_started":1}
I20260429 14:10:50.511843 26120 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44975 } has no permanent_uuid. Determining permanent_uuid...
Apr 29 14:10:50 dist-test-slave-8n77 krb5kdc[25882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.252@KRBTEST.COM
Apr 29 14:10:50 dist-test-slave-8n77 krb5kdc[25882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471850, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.252@KRBTEST.COM for kudu/127.2.197.253@KRBTEST.COM
I20260429 14:10:50.516458 26124 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.508261 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53396 (local address 127.2.197.252:46339)
0429 14:10:50.508429 (+ 168us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:50.508439 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:50.508526 (+ 87us) server_negotiation.cc:408] Connection header received
0429 14:10:50.508599 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:50.508602 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:50.508616 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:50.508692 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:50.509647 (+ 955us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.510642 (+ 995us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:50.511331 (+ 689us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.511547 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:50.513673 (+ 2126us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:50.513680 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:50.513681 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:50.513686 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:50.515342 (+ 1656us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:50.515708 (+ 366us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:50.515711 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:50.515711 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:50.515741 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:50.515925 (+ 184us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:50.515926 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:50.515926 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:50.516026 (+ 100us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:50.516169 (+ 143us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:50.516272 (+ 103us) server_negotiation.cc:300] Negotiation successful
0429 14:10:50.516348 (+ 76us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":85,"threads_started":1}
I20260429 14:10:50.518416 26047 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.512265 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:32894 (local address 127.2.197.253:44975)
0429 14:10:50.512373 (+ 108us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:50.512375 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:50.512382 (+ 7us) server_negotiation.cc:408] Connection header received
0429 14:10:50.512401 (+ 19us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:50.512402 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:50.512445 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:50.512513 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:50.512965 (+ 452us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.513472 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:50.514088 (+ 616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.514273 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:50.515752 (+ 1479us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:50.515762 (+ 10us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:50.515763 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:50.515768 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:50.517319 (+ 1551us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:50.517707 (+ 388us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:50.517708 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:50.517709 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:50.517734 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:50.517950 (+ 216us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:50.517951 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:50.517952 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:50.518070 (+ 118us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:50.518144 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:50.518232 (+ 88us) server_negotiation.cc:300] Negotiation successful
0429 14:10:50.518335 (+ 103us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260429 14:10:50.518436 26122 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.512132 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.253:44975 (local address 127.0.0.1:32894)
0429 14:10:50.512193 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:50.512205 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:50.512245 (+ 40us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:50.512518 (+ 273us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:50.512518 (+ 0us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:50.512687 (+ 169us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:10:50.512878 (+ 191us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:50.512880 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.513556 (+ 676us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:50.513557 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:50.514004 (+ 447us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:50.514006 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.514072 (+ 66us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:50.514499 (+ 427us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:10:50.514503 (+ 4us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:10:50.515660 (+ 1157us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:10:50.517455 (+ 1795us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:50.517456 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:50.517457 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:50.517641 (+ 184us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:50.517844 (+ 203us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:50.517845 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:50.517846 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:50.517874 (+ 28us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:50.518132 (+ 258us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:10:50.518133 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:10:50.518163 (+ 30us) client_negotiation.cc:770] Sending connection context
0429 14:10:50.518211 (+ 48us) client_negotiation.cc:241] Negotiation successful
0429 14:10:50.518356 (+ 145us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260429 14:10:50.518812 26120 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.197.252" port: 46339 } has no permanent_uuid. Determining permanent_uuid...
Apr 29 14:10:50 dist-test-slave-8n77 krb5kdc[25882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
Apr 29 14:10:50 dist-test-slave-8n77 krb5kdc[25882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471850, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.252@KRBTEST.COM for kudu/127.2.197.252@KRBTEST.COM
I20260429 14:10:50.524425 25969 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.517377 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60870 (local address 127.2.197.254:33447)
0429 14:10:50.517417 (+ 40us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:50.517419 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:50.517427 (+ 8us) server_negotiation.cc:408] Connection header received
0429 14:10:50.517479 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:50.517480 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:50.517491 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:50.517550 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:50.518366 (+ 816us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.518989 (+ 623us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:50.519611 (+ 622us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.519797 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:50.521852 (+ 2055us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:50.521856 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:50.521856 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:50.521862 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:50.523341 (+ 1479us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:50.523718 (+ 377us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:50.523719 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:50.523720 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:50.523744 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:50.523911 (+ 167us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:50.523912 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:50.523913 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:50.524002 (+ 89us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:50.524155 (+ 153us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:50.524234 (+ 79us) server_negotiation.cc:300] Negotiation successful
0429 14:10:50.524320 (+ 86us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260429 14:10:50.525024 26122 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.519089 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:46339 (local address 127.0.0.1:53404)
0429 14:10:50.519168 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:50.519179 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:50.519217 (+ 38us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:50.519350 (+ 133us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:50.519351 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:50.519464 (+ 113us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:10:50.519597 (+ 133us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:50.519600 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.520452 (+ 852us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:50.520453 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:50.520773 (+ 320us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:50.520774 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.520827 (+ 53us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:50.521199 (+ 372us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:10:50.521203 (+ 4us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:10:50.522520 (+ 1317us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:10:50.524054 (+ 1534us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:50.524055 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:50.524055 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:50.524242 (+ 187us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:50.524432 (+ 190us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:50.524435 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:50.524435 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:50.524502 (+ 67us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:50.524742 (+ 240us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:10:50.524743 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:10:50.524765 (+ 22us) client_negotiation.cc:770] Sending connection context
0429 14:10:50.524832 (+ 67us) client_negotiation.cc:241] Negotiation successful
0429 14:10:50.524916 (+ 84us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260429 14:10:50.525384 26124 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.519192 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53404 (local address 127.2.197.252:46339)
0429 14:10:50.519251 (+ 59us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:50.519252 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:50.519259 (+ 7us) server_negotiation.cc:408] Connection header received
0429 14:10:50.519276 (+ 17us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:50.519276 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:50.519287 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:50.519348 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:50.519683 (+ 335us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.520360 (+ 677us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:50.520851 (+ 491us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.521034 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:50.522631 (+ 1597us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:50.522634 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:50.522635 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:50.522641 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:50.523981 (+ 1340us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:50.524331 (+ 350us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:50.524332 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:50.524333 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:50.524356 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:50.524579 (+ 223us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:50.524580 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:50.524580 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:50.524667 (+ 87us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:50.524717 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:50.524880 (+ 163us) server_negotiation.cc:300] Negotiation successful
0429 14:10:50.524965 (+ 85us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":43}
I20260429 14:10:50.526891 26120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b19fb2f308f4439da00136d57a251789: Bootstrap starting.
I20260429 14:10:50.527967 26120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b19fb2f308f4439da00136d57a251789: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:50.528306 26120 log.cc:826] T 00000000000000000000000000000000 P b19fb2f308f4439da00136d57a251789: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:50.529316 26120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b19fb2f308f4439da00136d57a251789: No bootstrap required, opened a new log
Apr 29 14:10:50 dist-test-slave-8n77 krb5kdc[25882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.253@KRBTEST.COM
I20260429 14:10:50.530644 26120 raft_consensus.cc:359] T 00000000000000000000000000000000 P b19fb2f308f4439da00136d57a251789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9d971dc5f54ba3a3c23958a8e1ef99" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 33447 } } peers { permanent_uuid: "30459ec1650647c8a6158f0ef433d148" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44975 } } peers { permanent_uuid: "b19fb2f308f4439da00136d57a251789" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 46339 } }
I20260429 14:10:50.530778 26120 raft_consensus.cc:740] T 00000000000000000000000000000000 P b19fb2f308f4439da00136d57a251789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b19fb2f308f4439da00136d57a251789, State: Initialized, Role: FOLLOWER
I20260429 14:10:50.530867 26120 consensus_queue.cc:260] T 00000000000000000000000000000000 P b19fb2f308f4439da00136d57a251789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9d971dc5f54ba3a3c23958a8e1ef99" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 33447 } } peers { permanent_uuid: "30459ec1650647c8a6158f0ef433d148" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44975 } } peers { permanent_uuid: "b19fb2f308f4439da00136d57a251789" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 46339 } }
I20260429 14:10:50.531144 26126 sys_catalog.cc:455] T 00000000000000000000000000000000 P b19fb2f308f4439da00136d57a251789 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9d971dc5f54ba3a3c23958a8e1ef99" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 33447 } } peers { permanent_uuid: "30459ec1650647c8a6158f0ef433d148" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44975 } } peers { permanent_uuid: "b19fb2f308f4439da00136d57a251789" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 46339 } } }
I20260429 14:10:50.531196 26120 sys_catalog.cc:565] T 00000000000000000000000000000000 P b19fb2f308f4439da00136d57a251789 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:10:50.531311 26126 sys_catalog.cc:458] T 00000000000000000000000000000000 P b19fb2f308f4439da00136d57a251789 [sys.catalog]: This master's current role is: FOLLOWER
I20260429 14:10:50.533020 26047 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.525004 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:32904 (local address 127.2.197.253:44975)
0429 14:10:50.525042 (+ 38us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:50.525044 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:50.525052 (+ 8us) server_negotiation.cc:408] Connection header received
0429 14:10:50.525078 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:50.525079 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:50.525089 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:50.525148 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:50.525882 (+ 734us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.526422 (+ 540us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:50.527086 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.527206 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:50.530236 (+ 3030us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:50.530241 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:50.530242 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:50.530249 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:50.531864 (+ 1615us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:50.532209 (+ 345us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:50.532210 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:50.532210 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:50.532238 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:50.532388 (+ 150us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:50.532390 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:50.532390 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:50.532493 (+ 103us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:50.532637 (+ 144us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:50.532697 (+ 60us) server_negotiation.cc:300] Negotiation successful
0429 14:10:50.532787 (+ 90us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260429 14:10:50.533443 26137 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b19fb2f308f4439da00136d57a251789: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260429 14:10:50.533501 26137 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260429 14:10:50.534744 2839 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260429 14:10:50.533758 26052 master_runner.cc:186] Error getting master registration for 127.2.197.254:33447: OK, instance_id { permanent_uuid: "ff9d971dc5f54ba3a3c23958a8e1ef99" instance_seqno: 1777471850133881 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260429 14:10:50.535619 26052 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260429 14:10:50.535755 25940 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ff9d971dc5f54ba3a3c23958a8e1ef99"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60870
I20260429 14:10:50.554697 26043 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.197.252" port: 46339 } attempt: 3
Apr 29 14:10:50 dist-test-slave-8n77 krb5kdc[25882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471850, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.253@KRBTEST.COM for kudu/127.2.197.252@KRBTEST.COM
I20260429 14:10:50.562104 26045 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.555080 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:46339 (local address 127.0.0.1:53412)
0429 14:10:50.555138 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:50.555150 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:50.555195 (+ 45us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:50.555370 (+ 175us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:50.555371 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:50.555575 (+ 204us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:10:50.555781 (+ 206us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:50.555783 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.556737 (+ 954us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:50.556738 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:50.557289 (+ 551us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:50.557291 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.557369 (+ 78us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:50.557826 (+ 457us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:10:50.557831 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:10:50.559424 (+ 1593us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:10:50.561083 (+ 1659us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:50.561084 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:50.561085 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:50.561275 (+ 190us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:50.561461 (+ 186us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:50.561462 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:50.561462 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:50.561492 (+ 30us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:50.561776 (+ 284us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:10:50.561776 (+ 0us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:10:50.561824 (+ 48us) client_negotiation.cc:770] Sending connection context
0429 14:10:50.561896 (+ 72us) client_negotiation.cc:241] Negotiation successful
0429 14:10:50.561999 (+ 103us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260429 14:10:50.562130 26124 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.555154 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53412 (local address 127.2.197.252:46339)
0429 14:10:50.555216 (+ 62us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:50.555218 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:50.555235 (+ 17us) server_negotiation.cc:408] Connection header received
0429 14:10:50.555261 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:50.555262 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:50.555271 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:50.555334 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:50.555903 (+ 569us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.556645 (+ 742us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:50.557375 (+ 730us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.557551 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:50.559531 (+ 1980us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:50.559535 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:50.559536 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:50.559545 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:50.561003 (+ 1458us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:50.561359 (+ 356us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:50.561360 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:50.561361 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:50.561385 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:50.561549 (+ 164us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:50.561549 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:50.561550 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:50.561706 (+ 156us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:50.561772 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:50.561940 (+ 168us) server_negotiation.cc:300] Negotiation successful
0429 14:10:50.562033 (+ 93us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260429 14:10:50.563616 26043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30459ec1650647c8a6158f0ef433d148: Bootstrap starting.
I20260429 14:10:50.564534 26043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30459ec1650647c8a6158f0ef433d148: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:50.564805 26043 log.cc:826] T 00000000000000000000000000000000 P 30459ec1650647c8a6158f0ef433d148: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:50.565670 26043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30459ec1650647c8a6158f0ef433d148: No bootstrap required, opened a new log
I20260429 14:10:50.567334 26043 raft_consensus.cc:359] T 00000000000000000000000000000000 P 30459ec1650647c8a6158f0ef433d148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9d971dc5f54ba3a3c23958a8e1ef99" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 33447 } } peers { permanent_uuid: "30459ec1650647c8a6158f0ef433d148" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44975 } } peers { permanent_uuid: "b19fb2f308f4439da00136d57a251789" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 46339 } }
I20260429 14:10:50.567474 26043 raft_consensus.cc:740] T 00000000000000000000000000000000 P 30459ec1650647c8a6158f0ef433d148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30459ec1650647c8a6158f0ef433d148, State: Initialized, Role: FOLLOWER
I20260429 14:10:50.567540 26043 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30459ec1650647c8a6158f0ef433d148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9d971dc5f54ba3a3c23958a8e1ef99" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 33447 } } peers { permanent_uuid: "30459ec1650647c8a6158f0ef433d148" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44975 } } peers { permanent_uuid: "b19fb2f308f4439da00136d57a251789" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 46339 } }
I20260429 14:10:50.567814 26043 sys_catalog.cc:565] T 00000000000000000000000000000000 P 30459ec1650647c8a6158f0ef433d148 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:10:50.567817 26140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30459ec1650647c8a6158f0ef433d148 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9d971dc5f54ba3a3c23958a8e1ef99" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 33447 } } peers { permanent_uuid: "30459ec1650647c8a6158f0ef433d148" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44975 } } peers { permanent_uuid: "b19fb2f308f4439da00136d57a251789" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 46339 } } }
I20260429 14:10:50.567973 26140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30459ec1650647c8a6158f0ef433d148 [sys.catalog]: This master's current role is: FOLLOWER
W20260429 14:10:50.569756 26151 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 30459ec1650647c8a6158f0ef433d148: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260429 14:10:50.569819 26151 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260429 14:10:50.570174 25975 master_runner.cc:186] Error getting master registration for 127.2.197.254:33447: OK, instance_id { permanent_uuid: "ff9d971dc5f54ba3a3c23958a8e1ef99" instance_seqno: 1777471850133881 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260429 14:10:50.571856 25975 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260429 14:10:50.579779 25966 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.197.252" port: 46339 } attempt: 3
Apr 29 14:10:50 dist-test-slave-8n77 krb5kdc[25882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471850, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for kudu/127.2.197.252@KRBTEST.COM
I20260429 14:10:50.587698 26124 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.580257 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53426 (local address 127.2.197.252:46339)
0429 14:10:50.580299 (+ 42us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:50.580301 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:50.580312 (+ 11us) server_negotiation.cc:408] Connection header received
0429 14:10:50.580400 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:50.580402 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:50.580424 (+ 22us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:50.580478 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:50.581018 (+ 540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.581725 (+ 707us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:50.582692 (+ 967us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.582884 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:50.584695 (+ 1811us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:50.584699 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:50.584700 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:50.584707 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:50.586552 (+ 1845us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:50.586931 (+ 379us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:50.586932 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:50.586932 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:50.586957 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:50.587142 (+ 185us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:50.587143 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:50.587143 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:50.587259 (+ 116us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:50.587332 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:50.587513 (+ 181us) server_negotiation.cc:300] Negotiation successful
0429 14:10:50.587598 (+ 85us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260429 14:10:50.587698 25968 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.580186 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:46339 (local address 127.0.0.1:53426)
0429 14:10:50.580249 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:50.580261 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:50.580307 (+ 46us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:50.580507 (+ 200us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:50.580508 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:50.580699 (+ 191us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:10:50.580906 (+ 207us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:50.580908 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.581808 (+ 900us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:50.581808 (+ 0us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:50.582407 (+ 599us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:50.582409 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.582490 (+ 81us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:50.583005 (+ 515us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:10:50.583010 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:10:50.584601 (+ 1591us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:10:50.586651 (+ 2050us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:50.586652 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:50.586653 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:50.586853 (+ 200us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:50.587038 (+ 185us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:50.587038 (+ 0us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:50.587039 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:50.587081 (+ 42us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:50.587335 (+ 254us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:10:50.587336 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:10:50.587399 (+ 63us) client_negotiation.cc:770] Sending connection context
0429 14:10:50.587482 (+ 83us) client_negotiation.cc:241] Negotiation successful
0429 14:10:50.587589 (+ 107us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260429 14:10:50.589545 25966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff9d971dc5f54ba3a3c23958a8e1ef99: Bootstrap starting.
I20260429 14:10:50.590243 25966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff9d971dc5f54ba3a3c23958a8e1ef99: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:50.590583 25966 log.cc:826] T 00000000000000000000000000000000 P ff9d971dc5f54ba3a3c23958a8e1ef99: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:50.591459 25966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff9d971dc5f54ba3a3c23958a8e1ef99: No bootstrap required, opened a new log
I20260429 14:10:50.592485 25966 raft_consensus.cc:359] T 00000000000000000000000000000000 P ff9d971dc5f54ba3a3c23958a8e1ef99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9d971dc5f54ba3a3c23958a8e1ef99" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 33447 } } peers { permanent_uuid: "30459ec1650647c8a6158f0ef433d148" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44975 } } peers { permanent_uuid: "b19fb2f308f4439da00136d57a251789" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 46339 } }
I20260429 14:10:50.592613 25966 raft_consensus.cc:740] T 00000000000000000000000000000000 P ff9d971dc5f54ba3a3c23958a8e1ef99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff9d971dc5f54ba3a3c23958a8e1ef99, State: Initialized, Role: FOLLOWER
I20260429 14:10:50.592676 25966 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff9d971dc5f54ba3a3c23958a8e1ef99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9d971dc5f54ba3a3c23958a8e1ef99" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 33447 } } peers { permanent_uuid: "30459ec1650647c8a6158f0ef433d148" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44975 } } peers { permanent_uuid: "b19fb2f308f4439da00136d57a251789" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 46339 } }
I20260429 14:10:50.592960 25966 sys_catalog.cc:565] T 00000000000000000000000000000000 P ff9d971dc5f54ba3a3c23958a8e1ef99 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:10:50.594442 25940 raft_consensus.cc:493] T 00000000000000000000000000000000 P ff9d971dc5f54ba3a3c23958a8e1ef99 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260429 14:10:50.594524 25940 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ff9d971dc5f54ba3a3c23958a8e1ef99 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:10:50.595358 25940 raft_consensus.cc:515] T 00000000000000000000000000000000 P ff9d971dc5f54ba3a3c23958a8e1ef99 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9d971dc5f54ba3a3c23958a8e1ef99" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 33447 } } peers { permanent_uuid: "30459ec1650647c8a6158f0ef433d148" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44975 } } peers { permanent_uuid: "b19fb2f308f4439da00136d57a251789" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 46339 } }
I20260429 14:10:50.595772 25940 leader_election.cc:290] T 00000000000000000000000000000000 P ff9d971dc5f54ba3a3c23958a8e1ef99 [CANDIDATE]: Term 1 election: Requested vote from peers 30459ec1650647c8a6158f0ef433d148 (127.2.197.253:44975), b19fb2f308f4439da00136d57a251789 (127.2.197.252:46339)
I20260429 14:10:50.595983 26017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ff9d971dc5f54ba3a3c23958a8e1ef99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "30459ec1650647c8a6158f0ef433d148"
I20260429 14:10:50.596120 26017 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 30459ec1650647c8a6158f0ef433d148 [term 0 FOLLOWER]: Advancing to term 1
W20260429 14:10:50.596680 26165 catalog_manager.cc:1568] T 00000000000000000000000000000000 P ff9d971dc5f54ba3a3c23958a8e1ef99: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260429 14:10:50.596724 26165 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260429 14:10:50.596876 26154 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9d971dc5f54ba3a3c23958a8e1ef99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9d971dc5f54ba3a3c23958a8e1ef99" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 33447 } } peers { permanent_uuid: "30459ec1650647c8a6158f0ef433d148" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44975 } } peers { permanent_uuid: "b19fb2f308f4439da00136d57a251789" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 46339 } } }
I20260429 14:10:50.596938 26154 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9d971dc5f54ba3a3c23958a8e1ef99 [sys.catalog]: This master's current role is: FOLLOWER
I20260429 14:10:50.597054 26017 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 30459ec1650647c8a6158f0ef433d148 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff9d971dc5f54ba3a3c23958a8e1ef99 in term 1.
I20260429 14:10:50.597285 25914 leader_election.cc:304] T 00000000000000000000000000000000 P ff9d971dc5f54ba3a3c23958a8e1ef99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30459ec1650647c8a6158f0ef433d148, ff9d971dc5f54ba3a3c23958a8e1ef99; no voters:
I20260429 14:10:50.597321 26094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ff9d971dc5f54ba3a3c23958a8e1ef99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b19fb2f308f4439da00136d57a251789"
I20260429 14:10:50.597394 26154 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ff9d971dc5f54ba3a3c23958a8e1ef99 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:10:50.597420 26094 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b19fb2f308f4439da00136d57a251789 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:10:50.597513 26154 raft_consensus.cc:697] T 00000000000000000000000000000000 P ff9d971dc5f54ba3a3c23958a8e1ef99 [term 1 LEADER]: Becoming Leader. State: Replica: ff9d971dc5f54ba3a3c23958a8e1ef99, State: Running, Role: LEADER
I20260429 14:10:50.597589 26154 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff9d971dc5f54ba3a3c23958a8e1ef99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9d971dc5f54ba3a3c23958a8e1ef99" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 33447 } } peers { permanent_uuid: "30459ec1650647c8a6158f0ef433d148" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44975 } } peers { permanent_uuid: "b19fb2f308f4439da00136d57a251789" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 46339 } }
I20260429 14:10:50.597968 26154 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9d971dc5f54ba3a3c23958a8e1ef99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff9d971dc5f54ba3a3c23958a8e1ef99. Latest consensus state: current_term: 1 leader_uuid: "ff9d971dc5f54ba3a3c23958a8e1ef99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9d971dc5f54ba3a3c23958a8e1ef99" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 33447 } } peers { permanent_uuid: "30459ec1650647c8a6158f0ef433d148" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44975 } } peers { permanent_uuid: "b19fb2f308f4439da00136d57a251789" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 46339 } } }
I20260429 14:10:50.598038 26154 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9d971dc5f54ba3a3c23958a8e1ef99 [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:50.598183 26169 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:10:50.598270 26094 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b19fb2f308f4439da00136d57a251789 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff9d971dc5f54ba3a3c23958a8e1ef99 in term 1.
I20260429 14:10:50.598376 26169 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:10:50.600416 26094 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b19fb2f308f4439da00136d57a251789 [term 1 FOLLOWER]: Refusing update from remote peer ff9d971dc5f54ba3a3c23958a8e1ef99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:10:50.600644 26154 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ff9d971dc5f54ba3a3c23958a8e1ef99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19fb2f308f4439da00136d57a251789" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 46339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:10:50.600903 26017 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 30459ec1650647c8a6158f0ef433d148 [term 1 FOLLOWER]: Refusing update from remote peer ff9d971dc5f54ba3a3c23958a8e1ef99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:10:50.601099 26154 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ff9d971dc5f54ba3a3c23958a8e1ef99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30459ec1650647c8a6158f0ef433d148" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:10:50.601900 26126 sys_catalog.cc:455] T 00000000000000000000000000000000 P b19fb2f308f4439da00136d57a251789 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff9d971dc5f54ba3a3c23958a8e1ef99. Latest consensus state: current_term: 1 leader_uuid: "ff9d971dc5f54ba3a3c23958a8e1ef99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9d971dc5f54ba3a3c23958a8e1ef99" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 33447 } } peers { permanent_uuid: "30459ec1650647c8a6158f0ef433d148" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44975 } } peers { permanent_uuid: "b19fb2f308f4439da00136d57a251789" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 46339 } } }
I20260429 14:10:50.601969 26126 sys_catalog.cc:458] T 00000000000000000000000000000000 P b19fb2f308f4439da00136d57a251789 [sys.catalog]: This master's current role is: FOLLOWER
I20260429 14:10:50.601996 26140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30459ec1650647c8a6158f0ef433d148 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff9d971dc5f54ba3a3c23958a8e1ef99. Latest consensus state: current_term: 1 leader_uuid: "ff9d971dc5f54ba3a3c23958a8e1ef99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9d971dc5f54ba3a3c23958a8e1ef99" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 33447 } } peers { permanent_uuid: "30459ec1650647c8a6158f0ef433d148" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44975 } } peers { permanent_uuid: "b19fb2f308f4439da00136d57a251789" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 46339 } } }
I20260429 14:10:50.602093 26140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30459ec1650647c8a6158f0ef433d148 [sys.catalog]: This master's current role is: FOLLOWER
I20260429 14:10:50.603744 26170 mvcc.cc:204] Tried to move back new op lower bound from 7280524700055564288 to 7280524700048334848. Current Snapshot: MvccSnapshot[applied={T|T < 7280524700055564288}]
I20260429 14:10:50.603968 26154 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9d971dc5f54ba3a3c23958a8e1ef99 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ff9d971dc5f54ba3a3c23958a8e1ef99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9d971dc5f54ba3a3c23958a8e1ef99" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 33447 } } peers { permanent_uuid: "30459ec1650647c8a6158f0ef433d148" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44975 } } peers { permanent_uuid: "b19fb2f308f4439da00136d57a251789" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 46339 } } }
I20260429 14:10:50.604043 26154 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9d971dc5f54ba3a3c23958a8e1ef99 [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:50.604526 26126 sys_catalog.cc:455] T 00000000000000000000000000000000 P b19fb2f308f4439da00136d57a251789 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ff9d971dc5f54ba3a3c23958a8e1ef99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9d971dc5f54ba3a3c23958a8e1ef99" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 33447 } } peers { permanent_uuid: "30459ec1650647c8a6158f0ef433d148" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44975 } } peers { permanent_uuid: "b19fb2f308f4439da00136d57a251789" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 46339 } } }
I20260429 14:10:50.604597 26126 sys_catalog.cc:458] T 00000000000000000000000000000000 P b19fb2f308f4439da00136d57a251789 [sys.catalog]: This master's current role is: FOLLOWER
I20260429 14:10:50.604637 26154 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9d971dc5f54ba3a3c23958a8e1ef99 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ff9d971dc5f54ba3a3c23958a8e1ef99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9d971dc5f54ba3a3c23958a8e1ef99" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 33447 } } peers { permanent_uuid: "30459ec1650647c8a6158f0ef433d148" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44975 } } peers { permanent_uuid: "b19fb2f308f4439da00136d57a251789" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 46339 } } }
I20260429 14:10:50.604710 26154 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9d971dc5f54ba3a3c23958a8e1ef99 [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:50.604871 26175 mvcc.cc:204] Tried to move back new op lower bound from 7280524700055564288 to 7280524700048334848. Current Snapshot: MvccSnapshot[applied={T|T < 7280524700055564288}]
I20260429 14:10:50.605104 26140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30459ec1650647c8a6158f0ef433d148 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ff9d971dc5f54ba3a3c23958a8e1ef99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9d971dc5f54ba3a3c23958a8e1ef99" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 33447 } } peers { permanent_uuid: "30459ec1650647c8a6158f0ef433d148" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 44975 } } peers { permanent_uuid: "b19fb2f308f4439da00136d57a251789" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 46339 } } }
I20260429 14:10:50.605188 26140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30459ec1650647c8a6158f0ef433d148 [sys.catalog]: This master's current role is: FOLLOWER
I20260429 14:10:50.605367 26176 mvcc.cc:204] Tried to move back new op lower bound from 7280524700055564288 to 7280524700048334848. Current Snapshot: MvccSnapshot[applied={T|T < 7280524700055564288}]
I20260429 14:10:50.610932 26169 catalog_manager.cc:1357] Generated new cluster ID: 95c57eea6a9048219d8087b08216cba4
I20260429 14:10:50.610991 26169 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:10:50.613108 25969 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.602966 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60884 (local address 127.2.197.254:33447)
0429 14:10:50.603005 (+ 39us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:50.603007 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:50.605677 (+ 2670us) server_negotiation.cc:408] Connection header received
0429 14:10:50.605779 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:50.605780 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:50.605791 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:50.605820 (+ 29us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:50.606956 (+ 1136us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.607665 (+ 709us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:50.608419 (+ 754us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.608641 (+ 222us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:50.610144 (+ 1503us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:50.610147 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:50.610148 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:50.610155 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:50.612036 (+ 1881us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:50.612397 (+ 361us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:50.612398 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:50.612398 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:50.612418 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:50.612566 (+ 148us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:50.612567 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:50.612567 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:50.612669 (+ 102us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:50.612749 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:50.612934 (+ 185us) server_negotiation.cc:300] Negotiation successful
0429 14:10:50.613013 (+ 79us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260429 14:10:50.621100 26124 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.603343 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53428 (local address 127.2.197.252:46339)
0429 14:10:50.603377 (+ 34us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:50.603379 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:50.614076 (+ 10697us) server_negotiation.cc:408] Connection header received
0429 14:10:50.614169 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:50.614170 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:50.614181 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:50.614208 (+ 27us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:50.615005 (+ 797us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.615848 (+ 843us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:50.616572 (+ 724us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.616734 (+ 162us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:50.618463 (+ 1729us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:50.618468 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:50.618468 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:50.618475 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:50.620051 (+ 1576us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:50.620391 (+ 340us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:50.620392 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:50.620392 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:50.620414 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:50.620555 (+ 141us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:50.620556 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:50.620556 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:50.620660 (+ 104us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:50.620757 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:50.620922 (+ 165us) server_negotiation.cc:300] Negotiation successful
0429 14:10:50.620999 (+ 77us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260429 14:10:50.629400 26047 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:50.605449 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:32908 (local address 127.2.197.253:44975)
0429 14:10:50.605489 (+ 40us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:50.605492 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:50.622652 (+ 17160us) server_negotiation.cc:408] Connection header received
0429 14:10:50.622778 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:50.622779 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:50.622791 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:50.622823 (+ 32us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:50.623881 (+ 1058us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.624553 (+ 672us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:50.625193 (+ 640us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:50.625359 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:50.626728 (+ 1369us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:50.626732 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:50.626733 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:50.626739 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:50.628371 (+ 1632us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:50.628689 (+ 318us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:50.628690 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:50.628690 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:50.628711 (+ 21us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:50.628848 (+ 137us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:50.628849 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:50.628849 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:50.628951 (+ 102us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:50.629012 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:50.629178 (+ 166us) server_negotiation.cc:300] Negotiation successful
0429 14:10:50.629299 (+ 121us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260429 14:10:50.657577 26169 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:10:50.658143 26169 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:10:50.680620 26169 catalog_manager.cc:6044] T 00000000000000000000000000000000 P ff9d971dc5f54ba3a3c23958a8e1ef99: Generated new TSK 0
I20260429 14:10:50.680830 26169 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260429 14:10:50.702070 25969 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:10:50.700586 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60896 (local address 127.2.197.254:33447)
0429 14:10:50.700633 (+ 47us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:50.700636 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:50.701353 (+ 717us) server_negotiation.cc:408] Connection header received
0429 14:10:50.701468 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:50.701470 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:50.701481 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:50.701514 (+ 33us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:50.702003 (+ 489us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60896: BlockingRecv error: recv got EOF from 127.0.0.1:60896 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260429 14:10:50.702916 26124 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:10:50.701015 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53436 (local address 127.2.197.252:46339)
0429 14:10:50.701054 (+ 39us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:50.701056 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:50.702377 (+ 1321us) server_negotiation.cc:408] Connection header received
0429 14:10:50.702477 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:50.702478 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:50.702489 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:50.702519 (+ 30us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:50.702865 (+ 346us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53436: BlockingRecv error: recv got EOF from 127.0.0.1:53436 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260429 14:10:50.704120 26047 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:10:50.703321 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:32916 (local address 127.2.197.253:44975)
0429 14:10:50.703361 (+ 40us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:50.703364 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:50.703488 (+ 124us) server_negotiation.cc:408] Connection header received
0429 14:10:50.703600 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:50.703601 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:50.703612 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:50.703643 (+ 31us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:50.704068 (+ 425us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32916: BlockingRecv error: recv got EOF from 127.0.0.1:32916 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260429 14:10:50.706607 26047 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:10:50.705783 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:32922 (local address 127.2.197.253:44975)
0429 14:10:50.705818 (+ 35us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:50.705820 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:50.706016 (+ 196us) server_negotiation.cc:408] Connection header received
0429 14:10:50.706133 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:50.706134 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:50.706148 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:50.706178 (+ 30us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:50.706561 (+ 383us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32922: BlockingRecv error: recv got EOF from 127.0.0.1:32922 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260429 14:10:50.709755 26047 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:10:50.708963 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:32938 (local address 127.2.197.253:44975)
0429 14:10:50.708998 (+ 35us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:50.708999 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:50.709239 (+ 240us) server_negotiation.cc:408] Connection header received
0429 14:10:50.709341 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:50.709342 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:50.709353 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:50.709383 (+ 30us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:50.709711 (+ 328us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32938: BlockingRecv error: recv got EOF from 127.0.0.1:32938 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260429 14:10:50.715620 26047 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:10:50.714636 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:32948 (local address 127.2.197.253:44975)
0429 14:10:50.714686 (+ 50us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:50.714688 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:50.714851 (+ 163us) server_negotiation.cc:408] Connection header received
0429 14:10:50.714959 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:50.714960 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:50.714972 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:50.715003 (+ 31us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:50.715563 (+ 560us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32948: BlockingRecv error: recv got EOF from 127.0.0.1:32948 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260429 14:10:50.722039 26047 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:10:50.721093 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:32960 (local address 127.2.197.253:44975)
0429 14:10:50.721138 (+ 45us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:50.721140 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:50.721415 (+ 275us) server_negotiation.cc:408] Connection header received
0429 14:10:50.721542 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:50.721544 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:50.721556 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:50.721587 (+ 31us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:50.721981 (+ 394us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32960: BlockingRecv error: recv got EOF from 127.0.0.1:32960 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260429 14:10:50.732553 26047 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:10:50.731569 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:32972 (local address 127.2.197.253:44975)
0429 14:10:50.731624 (+ 55us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:50.731626 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:50.731925 (+ 299us) server_negotiation.cc:408] Connection header received
0429 14:10:50.732033 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:50.732034 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:50.732046 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:50.732077 (+ 31us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:50.732487 (+ 410us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32972: BlockingRecv error: recv got EOF from 127.0.0.1:32972 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260429 14:10:50.751148 26047 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:10:50.750176 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:32974 (local address 127.2.197.253:44975)
0429 14:10:50.750219 (+ 43us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:50.750222 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:50.750558 (+ 336us) server_negotiation.cc:408] Connection header received
0429 14:10:50.750668 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:50.750670 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:50.750681 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:50.750710 (+ 29us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:50.751091 (+ 381us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32974: BlockingRecv error: recv got EOF from 127.0.0.1:32974 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260429 14:10:50.785279 26047 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:10:50.784517 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:32986 (local address 127.2.197.253:44975)
0429 14:10:50.784566 (+ 49us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:50.784569 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:50.784733 (+ 164us) server_negotiation.cc:408] Connection header received
0429 14:10:50.784808 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:50.784809 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:50.784827 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:50.784889 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:50.785208 (+ 319us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32986: BlockingRecv error: recv got EOF from 127.0.0.1:32986 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260429 14:10:50.851709 26047 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:10:50.850755 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:32996 (local address 127.2.197.253:44975)
0429 14:10:50.850950 (+ 195us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:50.850953 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:50.851056 (+ 103us) server_negotiation.cc:408] Connection header received
0429 14:10:50.851157 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:50.851158 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:50.851170 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:50.851275 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:50.851626 (+ 351us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32996: BlockingRecv error: recv got EOF from 127.0.0.1:32996 (error 108)
Metrics: {"server-negotiator.queue_time_us":165}
W20260429 14:10:50.982327 26047 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:10:50.981253 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33008 (local address 127.2.197.253:44975)
0429 14:10:50.981341 (+ 88us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:50.981344 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:50.981430 (+ 86us) server_negotiation.cc:408] Connection header received
0429 14:10:50.981553 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:50.981554 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:50.981566 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:50.981645 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:50.982130 (+ 485us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33008: BlockingRecv error: recv got EOF from 127.0.0.1:33008 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260429 14:10:51.240458 26047 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:10:51.239762 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33016 (local address 127.2.197.253:44975)
0429 14:10:51.239865 (+ 103us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:51.239867 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:51.239878 (+ 11us) server_negotiation.cc:408] Connection header received
0429 14:10:51.239930 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:51.239931 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:51.239945 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:51.240018 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:51.240407 (+ 389us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33016: BlockingRecv error: recv got EOF from 127.0.0.1:33016 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260429 14:10:51.533922 26137 catalog_manager.cc:1269] Loaded cluster ID: 95c57eea6a9048219d8087b08216cba4
I20260429 14:10:51.534003 26137 catalog_manager.cc:1562] T 00000000000000000000000000000000 P b19fb2f308f4439da00136d57a251789: loading cluster ID for follower catalog manager: success
I20260429 14:10:51.534931 26137 catalog_manager.cc:1584] T 00000000000000000000000000000000 P b19fb2f308f4439da00136d57a251789: acquiring CA information for follower catalog manager: success
I20260429 14:10:51.535228 26137 catalog_manager.cc:1612] T 00000000000000000000000000000000 P b19fb2f308f4439da00136d57a251789: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260429 14:10:51.570262 26151 catalog_manager.cc:1269] Loaded cluster ID: 95c57eea6a9048219d8087b08216cba4
I20260429 14:10:51.570366 26151 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 30459ec1650647c8a6158f0ef433d148: loading cluster ID for follower catalog manager: success
I20260429 14:10:51.571273 26151 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 30459ec1650647c8a6158f0ef433d148: acquiring CA information for follower catalog manager: success
I20260429 14:10:51.571579 26151 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 30459ec1650647c8a6158f0ef433d148: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260429 14:10:51.756326 26249 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:51.753884 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33028 (local address 127.2.197.253:44975)
0429 14:10:51.754013 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:51.754016 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:51.754083 (+ 67us) server_negotiation.cc:408] Connection header received
0429 14:10:51.754205 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:51.754206 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:51.754256 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:51.754364 (+ 108us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:10:51.754704 (+ 340us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:51.755221 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:51.755806 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:51.755956 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:51.755986 (+ 30us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:10:51.756072 (+ 86us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:10:51.756167 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:51.756209 (+ 42us) server_negotiation.cc:300] Negotiation successful
0429 14:10:51.756234 (+ 25us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":55,"threads_started":1}
I20260429 14:10:51.760686 26255 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:51.758359 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53442 (local address 127.2.197.252:46339)
0429 14:10:51.758539 (+ 180us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:51.758541 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:51.758549 (+ 8us) server_negotiation.cc:408] Connection header received
0429 14:10:51.758612 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:51.758612 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:51.758639 (+ 27us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:51.758688 (+ 49us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:10:51.758923 (+ 235us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:51.759430 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:51.760119 (+ 689us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:51.760251 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:51.760285 (+ 34us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:10:51.760375 (+ 90us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:10:51.760436 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:51.760594 (+ 158us) server_negotiation.cc:300] Negotiation successful
0429 14:10:51.760622 (+ 28us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":134,"threads_started":1}
I20260429 14:10:51.765302 26255 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:51.762670 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53458 (local address 127.2.197.252:46339)
0429 14:10:51.762737 (+ 67us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:51.762739 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:51.762850 (+ 111us) server_negotiation.cc:408] Connection header received
0429 14:10:51.762905 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:51.762906 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:51.762920 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:51.762990 (+ 70us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:10:51.763352 (+ 362us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:51.763974 (+ 622us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:51.764665 (+ 691us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:51.764833 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:51.764944 (+ 111us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:10:51.764979 (+ 35us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:10:51.765011 (+ 32us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:51.765156 (+ 145us) server_negotiation.cc:300] Negotiation successful
0429 14:10:51.765189 (+ 33us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260429 14:10:51.765463 26249 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:51.762810 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33036 (local address 127.2.197.253:44975)
0429 14:10:51.762849 (+ 39us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:51.762851 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:51.763018 (+ 167us) server_negotiation.cc:408] Connection header received
0429 14:10:51.763093 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:51.763094 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:51.763109 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:51.763164 (+ 55us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:10:51.763544 (+ 380us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:51.764252 (+ 708us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:51.764970 (+ 718us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:51.765168 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:51.765191 (+ 23us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:10:51.765219 (+ 28us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:10:51.765271 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:51.765367 (+ 96us) server_negotiation.cc:300] Negotiation successful
0429 14:10:51.765396 (+ 29us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260429 14:10:51.766044 26263 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:51.762835 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60908 (local address 127.2.197.254:33447)
0429 14:10:51.762950 (+ 115us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:51.762952 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:51.763256 (+ 304us) server_negotiation.cc:408] Connection header received
0429 14:10:51.763313 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:51.763314 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:51.763342 (+ 28us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:51.763406 (+ 64us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:10:51.763708 (+ 302us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:51.764449 (+ 741us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:51.765063 (+ 614us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:51.765550 (+ 487us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:51.765606 (+ 56us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:10:51.765710 (+ 104us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:10:51.765780 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:51.765947 (+ 167us) server_negotiation.cc:300] Negotiation successful
0429 14:10:51.765975 (+ 28us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":67,"threads_started":1}
I20260429 14:10:51.768678 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 25898
I20260429 14:10:51.774171 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 25975
I20260429 14:10:51.779533 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 26052
2026-04-29T14:10:51Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3948 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 14:10:51 dist-test-slave-8n77 krb5kdc[26266](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 14:10:51 dist-test-slave-8n77 krb5kdc[26266](info): set up 2 sockets
Apr 29 14:10:51 dist-test-slave-8n77 krb5kdc[26266](info): commencing operation
krb5kdc: starting...
W20260429 14:10:53.835906 2839 mini_kdc.cc:121] Time spent starting KDC: real 2.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/krb5kdc/test-user.keytab.
Apr 29 14:10:53 dist-test-slave-8n77 krb5kdc[26266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T14:10:53Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T14:10:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:10:53.991417 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:35475
--webserver_interface=127.2.197.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:35535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:35475,127.2.197.253:46827,127.2.197.252:42465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 14:10:54.073550 26282 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260429 14:10:54.073815 26282 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:54.073848 26282 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:54.073890 26282 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:10:54.075536 26282 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:10:54.075630 26282 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:54.075655 26282 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:54.075676 26282 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:10:54.075707 26282 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:10:54.077445 26282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:35535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:35475,127.2.197.253:46827,127.2.197.252:42465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.254:35475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.197.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:54.077744 26282 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:54.077940 26282 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:10:54.080523 26287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:54.080523 26288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:54.080579 26290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:54.080686 26282 server_base.cc:1061] running on GCE node
I20260429 14:10:54.080883 26282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:54.081151 26282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:54.082358 26282 hybrid_clock.cc:648] HybridClock initialized: now 1777471854082330 us; error 35 us; skew 500 ppm
Apr 29 14:10:54 dist-test-slave-8n77 krb5kdc[26266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471854, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:10:54.085000 26282 init.cc:377] Logged in from keytab as kudu/127.2.197.254@KRBTEST.COM (short username kudu)
I20260429 14:10:54.085651 26282 webserver.cc:492] Webserver started at http://127.2.197.254:44799/ using document root <none> and password file <none>
I20260429 14:10:54.085850 26282 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:54.085912 26282 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:54.086019 26282 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:54.087363 26282 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-0/data/instance:
uuid: "192a0a5f3eed499b9a52d956b6a2d109"
format_stamp: "Formatted at 2026-04-29 14:10:54 on dist-test-slave-8n77"
server_key: "9e8c7747f45c3e25bdc7484e1efe856e"
server_key_iv: "4e35167d8ac14c89432d498455cb76d9"
server_key_version: "encryptionkey@0"
I20260429 14:10:54.087738 26282 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-0/wal/instance:
uuid: "192a0a5f3eed499b9a52d956b6a2d109"
format_stamp: "Formatted at 2026-04-29 14:10:54 on dist-test-slave-8n77"
server_key: "9e8c7747f45c3e25bdc7484e1efe856e"
server_key_iv: "4e35167d8ac14c89432d498455cb76d9"
server_key_version: "encryptionkey@0"
I20260429 14:10:54.089092 26282 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260429 14:10:54.090071 26297 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:54.090349 26282 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:10:54.090471 26282 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-0/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-0/wal
uuid: "192a0a5f3eed499b9a52d956b6a2d109"
format_stamp: "Formatted at 2026-04-29 14:10:54 on dist-test-slave-8n77"
server_key: "9e8c7747f45c3e25bdc7484e1efe856e"
server_key_iv: "4e35167d8ac14c89432d498455cb76d9"
server_key_version: "encryptionkey@0"
I20260429 14:10:54.090557 26282 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:54.102941 26282 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:54.104848 26282 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:54.105010 26282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:54.109915 26282 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.254:35475
I20260429 14:10:54.109982 26349 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.254:35475 every 8 connection(s)
I20260429 14:10:54.110338 26282 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-0/data/info.pb
I20260429 14:10:54.110955 26350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:54.112643 26350 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.197.254" port: 35475 } has no permanent_uuid. Determining permanent_uuid...
I20260429 14:10:54.116451 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 26282
I20260429 14:10:54.116640 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-0/wal/instance
I20260429 14:10:54.116863 2839 external_mini_cluster.cc:1468] Setting key b4a65d6dde76140f97ed626434d4af44
Apr 29 14:10:54 dist-test-slave-8n77 krb5kdc[26266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471854, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:10:54.122927 26352 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.113262 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:35475 (local address 127.0.0.1:54384)
0429 14:10:54.113426 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:54.113485 (+ 59us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:54.113848 (+ 363us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:54.114098 (+ 250us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:54.114100 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:54.114508 (+ 408us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:10:54.114739 (+ 231us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:54.114743 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.115609 (+ 866us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:54.115611 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:54.115965 (+ 354us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:54.115966 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.116127 (+ 161us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:54.116899 (+ 772us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:10:54.116919 (+ 20us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:10:54.119394 (+ 2475us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:10:54.121649 (+ 2255us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:54.121654 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:54.121656 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:54.121830 (+ 174us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:54.122057 (+ 227us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:54.122057 (+ 0us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:54.122058 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:54.122102 (+ 44us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:54.122552 (+ 450us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:10:54.122556 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:10:54.122608 (+ 52us) client_negotiation.cc:770] Sending connection context
0429 14:10:54.122685 (+ 77us) client_negotiation.cc:241] Negotiation successful
0429 14:10:54.122788 (+ 103us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":91,"threads_started":1}
I20260429 14:10:54.123020 26353 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.113381 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54384 (local address 127.2.197.254:35475)
0429 14:10:54.113541 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:54.113547 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:54.113569 (+ 22us) server_negotiation.cc:408] Connection header received
0429 14:10:54.113985 (+ 416us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:54.113987 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:54.114008 (+ 21us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:54.114093 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:54.114843 (+ 750us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.115524 (+ 681us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:54.116047 (+ 523us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.116226 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:54.119513 (+ 3287us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:54.119529 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:54.119529 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:54.119536 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:54.121544 (+ 2008us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:54.121921 (+ 377us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:54.121925 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:54.121926 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:54.121984 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:54.122168 (+ 184us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:54.122169 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:54.122170 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:54.122482 (+ 312us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:54.122579 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:54.122744 (+ 165us) server_negotiation.cc:300] Negotiation successful
0429 14:10:54.122860 (+ 116us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":135,"thread_start_us":87,"threads_started":1}
I20260429 14:10:54.123790 26350 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.197.253" port: 46827 } has no permanent_uuid. Determining permanent_uuid...
I20260429 14:10:54.124181 26352 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.124063 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.253:46827 (local address 127.0.0.1:34084)
0429 14:10:54.124115 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:54.124162 (+ 47us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.253:46827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260429 14:10:54.124325 26299 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.197.253:46827: connect: Connection refused (error 111)
I20260429 14:10:54.125030 26352 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.124909 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.253:46827 (local address 127.0.0.1:34094)
0429 14:10:54.124981 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:54.125013 (+ 32us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.253:46827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260429 14:10:54.125164 26350 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.197.253:46827: Network error: Client connection negotiation failed: client connection to 127.2.197.253:46827: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.2.197.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.197.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.253@KRBTEST.COM" created.
I20260429 14:10:54.174410 26350 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.197.253" port: 46827 } attempt: 1
I20260429 14:10:54.174959 26352 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.174830 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.253:46827 (local address 127.0.0.1:34104)
0429 14:10:54.174890 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:54.174929 (+ 39us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.253:46827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260429 14:10:54.175766 26352 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.175630 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.253:46827 (local address 127.0.0.1:34108)
0429 14:10:54.175708 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:54.175743 (+ 35us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.253:46827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260429 14:10:54.175944 26350 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.197.253:46827: Network error: Client connection negotiation failed: client connection to 127.2.197.253:46827: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:10:54.179351 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.253:46827
--webserver_interface=127.2.197.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:35535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:35475,127.2.197.253:46827,127.2.197.252:42465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260429 14:10:54.244153 26350 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.197.253" port: 46827 } attempt: 2
I20260429 14:10:54.244784 26352 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.244634 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.253:46827 (local address 127.0.0.1:34112)
0429 14:10:54.244712 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:54.244753 (+ 41us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.253:46827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260429 14:10:54.245257 26352 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.245132 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.253:46827 (local address 127.0.0.1:34118)
0429 14:10:54.245201 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:54.245233 (+ 32us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.253:46827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260429 14:10:54.245416 26350 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.197.253:46827: Network error: Client connection negotiation failed: client connection to 127.2.197.253:46827: connect: Connection refused (error 111)
W20260429 14:10:54.259034 26359 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260429 14:10:54.259284 26359 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:54.259315 26359 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:54.259380 26359 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:10:54.260958 26359 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:10:54.261040 26359 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:54.261065 26359 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:54.261099 26359 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:10:54.261126 26359 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:10:54.262918 26359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:35535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:35475,127.2.197.253:46827,127.2.197.252:42465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.253:46827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.197.253
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:54.263200 26359 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:54.263389 26359 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:10:54.265908 26367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:54.266062 26365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:54.265898 26364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:54.266331 26359 server_base.cc:1061] running on GCE node
I20260429 14:10:54.266554 26359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:54.266798 26359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:54.267973 26359 hybrid_clock.cc:648] HybridClock initialized: now 1777471854267943 us; error 35 us; skew 500 ppm
Apr 29 14:10:54 dist-test-slave-8n77 krb5kdc[26266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471854, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:10:54.270543 26359 init.cc:377] Logged in from keytab as kudu/127.2.197.253@KRBTEST.COM (short username kudu)
I20260429 14:10:54.271064 26359 webserver.cc:492] Webserver started at http://127.2.197.253:45231/ using document root <none> and password file <none>
I20260429 14:10:54.271278 26359 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:54.271337 26359 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:54.271440 26359 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:54.272387 26359 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-1/data/instance:
uuid: "d7064b775fdd4fe5bc1eccd89d6764cb"
format_stamp: "Formatted at 2026-04-29 14:10:54 on dist-test-slave-8n77"
server_key: "bfc1f30b51ba2321c8296a2d16dcd1a4"
server_key_iv: "b7abcde13fd6e383bf8ecda58671e9ad"
server_key_version: "encryptionkey@0"
I20260429 14:10:54.272727 26359 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-1/wal/instance:
uuid: "d7064b775fdd4fe5bc1eccd89d6764cb"
format_stamp: "Formatted at 2026-04-29 14:10:54 on dist-test-slave-8n77"
server_key: "bfc1f30b51ba2321c8296a2d16dcd1a4"
server_key_iv: "b7abcde13fd6e383bf8ecda58671e9ad"
server_key_version: "encryptionkey@0"
I20260429 14:10:54.274039 26359 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260429 14:10:54.274932 26374 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:54.275293 26359 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 14:10:54.275380 26359 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-1/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-1/wal
uuid: "d7064b775fdd4fe5bc1eccd89d6764cb"
format_stamp: "Formatted at 2026-04-29 14:10:54 on dist-test-slave-8n77"
server_key: "bfc1f30b51ba2321c8296a2d16dcd1a4"
server_key_iv: "b7abcde13fd6e383bf8ecda58671e9ad"
server_key_version: "encryptionkey@0"
I20260429 14:10:54.275449 26359 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:54.287506 26359 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:54.289345 26359 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:54.289506 26359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:54.293566 26359 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.253:46827
I20260429 14:10:54.293604 26426 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.253:46827 every 8 connection(s)
I20260429 14:10:54.293936 26359 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-1/data/info.pb
I20260429 14:10:54.294214 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 26359
I20260429 14:10:54.294656 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-1/wal/instance
I20260429 14:10:54.294960 2839 external_mini_cluster.cc:1468] Setting key 95ebd9217b90090be20340073cf6fb8e
I20260429 14:10:54.295147 26427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:54.297688 26427 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.197.254" port: 35475 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.197.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.197.252@KRBTEST.COM" created.
Apr 29 14:10:54 dist-test-slave-8n77 krb5kdc[26266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471854, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.253@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:10:54.310019 26430 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.298522 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:35475 (local address 127.0.0.1:54390)
0429 14:10:54.298712 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:54.298743 (+ 31us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:54.299159 (+ 416us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:54.299392 (+ 233us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:54.299398 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:54.299701 (+ 303us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:10:54.299962 (+ 261us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:54.299965 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.300657 (+ 692us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:54.300660 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:54.301023 (+ 363us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:54.301024 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.301163 (+ 139us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:54.301852 (+ 689us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:10:54.301862 (+ 10us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:10:54.306531 (+ 4669us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:10:54.308739 (+ 2208us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:54.308745 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:54.308747 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:54.308965 (+ 218us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:54.309197 (+ 232us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:54.309198 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:54.309198 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:54.309261 (+ 63us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:54.309554 (+ 293us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:10:54.309559 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:10:54.309674 (+ 115us) client_negotiation.cc:770] Sending connection context
0429 14:10:54.309767 (+ 93us) client_negotiation.cc:241] Negotiation successful
0429 14:10:54.309874 (+ 107us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":95,"threads_started":1}
I20260429 14:10:54.310048 26353 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.298539 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54390 (local address 127.2.197.254:35475)
0429 14:10:54.298603 (+ 64us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:54.298607 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:54.298823 (+ 216us) server_negotiation.cc:408] Connection header received
0429 14:10:54.299261 (+ 438us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:54.299262 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:54.299277 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:54.299340 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:54.300065 (+ 725us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.300553 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:54.301105 (+ 552us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.301283 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:54.306670 (+ 5387us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:54.306675 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:54.306675 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:54.306683 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:54.308614 (+ 1931us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:54.309059 (+ 445us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:54.309061 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:54.309061 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:54.309096 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:54.309332 (+ 236us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:54.309333 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:54.309334 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:54.309476 (+ 142us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:54.309558 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:54.309817 (+ 259us) server_negotiation.cc:300] Negotiation successful
0429 14:10:54.309928 (+ 111us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260429 14:10:54.310711 26427 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.197.253" port: 46827 } has no permanent_uuid. Determining permanent_uuid...
Apr 29 14:10:54 dist-test-slave-8n77 krb5kdc[26266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471854, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.253@KRBTEST.COM for kudu/127.2.197.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.197.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.197.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:10:54.326822 26430 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.311080 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.253:46827 (local address 127.0.0.1:34134)
0429 14:10:54.311140 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:54.311153 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:54.311201 (+ 48us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:54.311522 (+ 321us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:54.311523 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:54.311693 (+ 170us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:10:54.311902 (+ 209us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:54.311905 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.312795 (+ 890us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:54.312796 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:54.313374 (+ 578us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:54.313376 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.313473 (+ 97us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:54.313954 (+ 481us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:10:54.313959 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:10:54.317701 (+ 3742us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:10:54.325701 (+ 8000us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:54.325703 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:54.325704 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:54.325928 (+ 224us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:54.326124 (+ 196us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:54.326125 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:54.326126 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:54.326162 (+ 36us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:54.326463 (+ 301us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:10:54.326464 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:10:54.326517 (+ 53us) client_negotiation.cc:770] Sending connection context
0429 14:10:54.326601 (+ 84us) client_negotiation.cc:241] Negotiation successful
0429 14:10:54.326738 (+ 137us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260429 14:10:54.326974 26432 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.311186 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34134 (local address 127.2.197.253:46827)
0429 14:10:54.311367 (+ 181us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:54.311373 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:54.311388 (+ 15us) server_negotiation.cc:408] Connection header received
0429 14:10:54.311406 (+ 18us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:54.311408 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:54.311419 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:54.311485 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:54.311994 (+ 509us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.312701 (+ 707us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:54.313495 (+ 794us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.313734 (+ 239us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:54.317838 (+ 4104us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:54.317848 (+ 10us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:54.317848 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:54.317855 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:54.325579 (+ 7724us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:54.326027 (+ 448us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:54.326030 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:54.326030 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:54.326054 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:54.326230 (+ 176us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:54.326231 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:54.326231 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:54.326398 (+ 167us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:54.326464 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:54.326662 (+ 198us) server_negotiation.cc:300] Negotiation successful
0429 14:10:54.326844 (+ 182us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":140,"thread_start_us":111,"threads_started":1}
I20260429 14:10:54.327567 26427 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42465 } has no permanent_uuid. Determining permanent_uuid...
I20260429 14:10:54.327970 26430 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.327832 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:42465 (local address 127.0.0.1:52620)
0429 14:10:54.327883 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:54.327937 (+ 54us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.252:42465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260429 14:10:54.328096 26376 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.197.252:42465: connect: Connection refused (error 111)
I20260429 14:10:54.328909 26430 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.328670 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:42465 (local address 127.0.0.1:52634)
0429 14:10:54.328834 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:54.328873 (+ 39us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.252:42465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
W20260429 14:10:54.329118 26427 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.197.252:42465: Network error: Client connection negotiation failed: client connection to 127.2.197.252:42465: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.2.197.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.197.252@KRBTEST.COM" created.
I20260429 14:10:54.336580 26350 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.197.253" port: 46827 } attempt: 3
Apr 29 14:10:54 dist-test-slave-8n77 krb5kdc[26266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471854, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for kudu/127.2.197.253@KRBTEST.COM
I20260429 14:10:54.343971 26352 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.336955 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.253:46827 (local address 127.0.0.1:34148)
0429 14:10:54.337029 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:54.337044 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:54.337130 (+ 86us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:54.337333 (+ 203us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:54.337334 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:54.337503 (+ 169us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:10:54.337698 (+ 195us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:54.337700 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.338405 (+ 705us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:54.338406 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:54.338785 (+ 379us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:54.338787 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.338875 (+ 88us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:54.339449 (+ 574us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:10:54.339453 (+ 4us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:10:54.340926 (+ 1473us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:10:54.342795 (+ 1869us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:54.342796 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:54.342797 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:54.342950 (+ 153us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:54.343289 (+ 339us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:54.343290 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:54.343290 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:54.343329 (+ 39us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:54.343603 (+ 274us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:10:54.343604 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:10:54.343666 (+ 62us) client_negotiation.cc:770] Sending connection context
0429 14:10:54.343756 (+ 90us) client_negotiation.cc:241] Negotiation successful
0429 14:10:54.343872 (+ 116us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260429 14:10:54.343971 26432 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.337076 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34148 (local address 127.2.197.253:46827)
0429 14:10:54.337200 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:54.337203 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:54.337217 (+ 14us) server_negotiation.cc:408] Connection header received
0429 14:10:54.337240 (+ 23us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:54.337240 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:54.337251 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:54.337328 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:54.337784 (+ 456us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.338248 (+ 464us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:54.338868 (+ 620us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.339072 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:54.341026 (+ 1954us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:54.341037 (+ 11us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:54.341038 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:54.341044 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:54.342692 (+ 1648us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:54.343038 (+ 346us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:54.343039 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:54.343040 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:54.343072 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:54.343397 (+ 325us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:54.343398 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:54.343399 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:54.343515 (+ 116us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:54.343612 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:54.343783 (+ 171us) server_negotiation.cc:300] Negotiation successful
0429 14:10:54.343871 (+ 88us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260429 14:10:54.344580 26350 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42465 } has no permanent_uuid. Determining permanent_uuid...
I20260429 14:10:54.345026 26352 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.344923 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:42465 (local address 127.0.0.1:52648)
0429 14:10:54.344973 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:54.345006 (+ 33us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.252:42465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260429 14:10:54.345378 26352 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.345285 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:42465 (local address 127.0.0.1:52664)
0429 14:10:54.345332 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:54.345362 (+ 30us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.252:42465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260429 14:10:54.345474 26350 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.197.252:42465: Network error: Client connection negotiation failed: client connection to 127.2.197.252:42465: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.197.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.197.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/krb5kdc/kudu.keytab.
I20260429 14:10:54.353274 2839 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskhpfkRA/build/release/bin/kudu
/tmp/dist-test-taskhpfkRA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.252:42465
--webserver_interface=127.2.197.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.197.212:35535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.197.254:35475,127.2.197.253:46827,127.2.197.252:42465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260429 14:10:54.376638 26350 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42465 } attempt: 1
I20260429 14:10:54.377170 26352 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.377006 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:42465 (local address 127.0.0.1:52676)
0429 14:10:54.377076 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:54.377132 (+ 56us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.252:42465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260429 14:10:54.377624 26352 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.377511 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:42465 (local address 127.0.0.1:52678)
0429 14:10:54.377567 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:54.377600 (+ 33us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.252:42465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260429 14:10:54.377745 26350 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.197.252:42465: Network error: Client connection negotiation failed: client connection to 127.2.197.252:42465: connect: Connection refused (error 111)
I20260429 14:10:54.378347 26427 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42465 } attempt: 1
I20260429 14:10:54.378777 26430 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.378662 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:42465 (local address 127.0.0.1:52684)
0429 14:10:54.378717 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:54.378751 (+ 34us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.252:42465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260429 14:10:54.379098 26430 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.379003 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:42465 (local address 127.0.0.1:52696)
0429 14:10:54.379053 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:54.379080 (+ 27us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.252:42465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260429 14:10:54.379242 26427 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.197.252:42465: Network error: Client connection negotiation failed: client connection to 127.2.197.252:42465: connect: Connection refused (error 111)
W20260429 14:10:54.431510 26436 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260429 14:10:54.431726 26436 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 14:10:54.431743 26436 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 14:10:54.431766 26436 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 14:10:54.433367 26436 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 14:10:54.433449 26436 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 14:10:54.433465 26436 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 14:10:54.433477 26436 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 14:10:54.433490 26436 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 14:10:54.435292 26436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.197.212:35535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.197.254:35475,127.2.197.253:46827,127.2.197.252:42465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.197.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.197.252:42465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.197.252
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type RELEASE
built by None at 29 Apr 2026 13:43:15 UTC on e7f111948823
build id 11661
I20260429 14:10:54.435556 26436 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 14:10:54.435756 26436 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 14:10:54.438385 26441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:54.438431 26442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 14:10:54.438452 26444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 14:10:54.438498 26436 server_base.cc:1061] running on GCE node
I20260429 14:10:54.438813 26436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 14:10:54.439056 26436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 14:10:54.440248 26436 hybrid_clock.cc:648] HybridClock initialized: now 1777471854440274 us; error 111 us; skew 500 ppm
Apr 29 14:10:54 dist-test-slave-8n77 krb5kdc[26266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471854, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 14:10:54.442834 26436 init.cc:377] Logged in from keytab as kudu/127.2.197.252@KRBTEST.COM (short username kudu)
I20260429 14:10:54.443421 26436 webserver.cc:492] Webserver started at http://127.2.197.252:33869/ using document root <none> and password file <none>
I20260429 14:10:54.443627 26436 fs_manager.cc:362] Metadata directory not provided
I20260429 14:10:54.443670 26436 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 14:10:54.443874 26436 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 14:10:54.444898 26436 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-2/data/instance:
uuid: "62dad35d86e44387be5e07fcb6f3a6c0"
format_stamp: "Formatted at 2026-04-29 14:10:54 on dist-test-slave-8n77"
server_key: "2dd54ac0de740096bb8228ae0b4c7ae6"
server_key_iv: "44dbfaf005aee42de3a4d9905a849b95"
server_key_version: "encryptionkey@0"
I20260429 14:10:54.445351 26436 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-2/wal/instance:
uuid: "62dad35d86e44387be5e07fcb6f3a6c0"
format_stamp: "Formatted at 2026-04-29 14:10:54 on dist-test-slave-8n77"
server_key: "2dd54ac0de740096bb8228ae0b4c7ae6"
server_key_iv: "44dbfaf005aee42de3a4d9905a849b95"
server_key_version: "encryptionkey@0"
I20260429 14:10:54.446717 26436 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260429 14:10:54.447475 26427 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42465 } attempt: 2
I20260429 14:10:54.447703 26451 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 14:10:54.448084 26430 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.447828 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:42465 (local address 127.0.0.1:52710)
0429 14:10:54.448008 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:54.448050 (+ 42us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.252:42465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260429 14:10:54.448240 26436 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 14:10:54.448312 26436 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-2/data,/tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-2/wal
uuid: "62dad35d86e44387be5e07fcb6f3a6c0"
format_stamp: "Formatted at 2026-04-29 14:10:54 on dist-test-slave-8n77"
server_key: "2dd54ac0de740096bb8228ae0b4c7ae6"
server_key_iv: "44dbfaf005aee42de3a4d9905a849b95"
server_key_version: "encryptionkey@0"
I20260429 14:10:54.448376 26436 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 14:10:54.448542 26430 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.448444 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:42465 (local address 127.0.0.1:52718)
0429 14:10:54.448479 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:54.448508 (+ 29us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.252:42465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260429 14:10:54.448657 26427 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.197.252:42465: Network error: Client connection negotiation failed: client connection to 127.2.197.252:42465: connect: Connection refused (error 111)
I20260429 14:10:54.452891 26350 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42465 } attempt: 2
I20260429 14:10:54.453323 26352 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.453174 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:42465 (local address 127.0.0.1:52728)
0429 14:10:54.453258 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:54.453295 (+ 37us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.252:42465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260429 14:10:54.453667 26352 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.453571 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:42465 (local address 127.0.0.1:52740)
0429 14:10:54.453617 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:54.453643 (+ 26us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.197.252:42465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260429 14:10:54.453754 26350 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.197.252:42465: Network error: Client connection negotiation failed: client connection to 127.2.197.252:42465: connect: Connection refused (error 111)
I20260429 14:10:54.465381 26436 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 14:10:54.467487 26436 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 14:10:54.467655 26436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 14:10:54.471779 26436 rpc_server.cc:307] RPC server started. Bound to: 127.2.197.252:42465
I20260429 14:10:54.471812 26503 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.197.252:42465 every 8 connection(s)
I20260429 14:10:54.472121 26436 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-2/data/info.pb
I20260429 14:10:54.472654 26504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 14:10:54.474472 26504 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.197.254" port: 35475 } has no permanent_uuid. Determining permanent_uuid...
I20260429 14:10:54.478418 2839 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskhpfkRA/build/release/bin/kudu as pid 26436
I20260429 14:10:54.478543 2839 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskhpfkRA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777471413505459-2839-0/minicluster-data/master-2/wal/instance
I20260429 14:10:54.478715 2839 external_mini_cluster.cc:1468] Setting key 07ff60eaf45e2abc91a80284216650cc
Apr 29 14:10:54 dist-test-slave-8n77 krb5kdc[26266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471854, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.252@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
Apr 29 14:10:54 dist-test-slave-8n77 krb5kdc[26266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.252@KRBTEST.COM
I20260429 14:10:54.483579 26353 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.475283 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54394 (local address 127.2.197.254:35475)
0429 14:10:54.475399 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:54.475402 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:54.475566 (+ 164us) server_negotiation.cc:408] Connection header received
0429 14:10:54.475957 (+ 391us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:54.475958 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:54.475971 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:54.476023 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:54.476754 (+ 731us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.477189 (+ 435us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:54.477916 (+ 727us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.478095 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:54.480508 (+ 2413us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:54.480512 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:54.480513 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:54.480520 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:54.482102 (+ 1582us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:54.482643 (+ 541us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:54.482645 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:54.482645 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:54.482671 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:54.482893 (+ 222us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:54.482894 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:54.482895 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:54.482996 (+ 101us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:54.483074 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:54.483384 (+ 310us) server_negotiation.cc:300] Negotiation successful
0429 14:10:54.483485 (+ 101us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260429 14:10:54.483752 26506 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.475151 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.254:35475 (local address 127.0.0.1:54394)
0429 14:10:54.475430 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:54.475482 (+ 52us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:54.475858 (+ 376us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:54.476098 (+ 240us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:54.476102 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:54.476391 (+ 289us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:10:54.476654 (+ 263us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:54.476657 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.477446 (+ 789us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:54.477447 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:54.477829 (+ 382us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:54.477830 (+ 1us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.477969 (+ 139us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:54.478666 (+ 697us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:10:54.478679 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:10:54.480378 (+ 1699us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:10:54.482306 (+ 1928us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:54.482313 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:54.482315 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:54.482544 (+ 229us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:54.482749 (+ 205us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:54.482750 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:54.482751 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:54.482823 (+ 72us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:54.483064 (+ 241us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:10:54.483069 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:10:54.483247 (+ 178us) client_negotiation.cc:770] Sending connection context
0429 14:10:54.483366 (+ 119us) client_negotiation.cc:241] Negotiation successful
0429 14:10:54.483498 (+ 132us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":93,"threads_started":1}
I20260429 14:10:54.484391 26504 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.197.253" port: 46827 } has no permanent_uuid. Determining permanent_uuid...
I20260429 14:10:54.487411 26508 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.479132 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:52754 (local address 127.2.197.252:42465)
0429 14:10:54.479304 (+ 172us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:54.479312 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:54.479416 (+ 104us) server_negotiation.cc:408] Connection header received
0429 14:10:54.479485 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:54.479488 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:54.479507 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:54.479582 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:54.480458 (+ 876us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.481121 (+ 663us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:54.481885 (+ 764us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.482073 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:54.484079 (+ 2006us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:54.484088 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:54.484089 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:54.484096 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:54.485808 (+ 1712us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:54.486169 (+ 361us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:54.486172 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:54.486178 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:54.486207 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:54.486672 (+ 465us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:54.486673 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:54.486673 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:54.486803 (+ 130us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:54.486997 (+ 194us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:54.487046 (+ 49us) server_negotiation.cc:300] Negotiation successful
0429 14:10:54.487150 (+ 104us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":89,"threads_started":1}
Apr 29 14:10:54 dist-test-slave-8n77 krb5kdc[26266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471854, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.252@KRBTEST.COM for kudu/127.2.197.253@KRBTEST.COM
I20260429 14:10:54.491494 26432 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.484742 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34156 (local address 127.2.197.253:46827)
0429 14:10:54.484812 (+ 70us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:54.484814 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:54.484822 (+ 8us) server_negotiation.cc:408] Connection header received
0429 14:10:54.484844 (+ 22us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:54.484845 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:54.484855 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:54.484927 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:54.485303 (+ 376us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.485733 (+ 430us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:54.486404 (+ 671us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.486614 (+ 210us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:54.488616 (+ 2002us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:54.488621 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:54.488622 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:54.488629 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:54.490402 (+ 1773us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:54.490788 (+ 386us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:54.490790 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:54.490790 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:54.490822 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:54.490990 (+ 168us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:54.490991 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:54.490992 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:54.491093 (+ 101us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:54.491170 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:54.491314 (+ 144us) server_negotiation.cc:300] Negotiation successful
0429 14:10:54.491397 (+ 83us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260429 14:10:54.491503 26506 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.484660 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.253:46827 (local address 127.0.0.1:34156)
0429 14:10:54.484710 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:54.484720 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:54.484757 (+ 37us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:54.484930 (+ 173us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:54.484931 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:54.485087 (+ 156us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:10:54.485221 (+ 134us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:54.485223 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.485815 (+ 592us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:54.485816 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:54.486211 (+ 395us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:54.486213 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.486379 (+ 166us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:54.486801 (+ 422us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:10:54.486805 (+ 4us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:10:54.488533 (+ 1728us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:10:54.490494 (+ 1961us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:54.490495 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:54.490496 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:54.490686 (+ 190us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:54.490897 (+ 211us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:54.490898 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:54.490898 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:54.490930 (+ 32us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:54.491159 (+ 229us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:10:54.491159 (+ 0us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:10:54.491206 (+ 47us) client_negotiation.cc:770] Sending connection context
0429 14:10:54.491289 (+ 83us) client_negotiation.cc:241] Negotiation successful
0429 14:10:54.491395 (+ 106us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260429 14:10:54.491851 26504 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42465 } has no permanent_uuid. Determining permanent_uuid...
Apr 29 14:10:54 dist-test-slave-8n77 krb5kdc[26266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.254@KRBTEST.COM
I20260429 14:10:54.495328 26353 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.488208 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54396 (local address 127.2.197.254:35475)
0429 14:10:54.488258 (+ 50us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:54.488260 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:54.488272 (+ 12us) server_negotiation.cc:408] Connection header received
0429 14:10:54.488341 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:54.488342 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:54.488353 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:54.488421 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:54.489030 (+ 609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.489591 (+ 561us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:54.490378 (+ 787us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.490580 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:54.492667 (+ 2087us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:54.492671 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:54.492672 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:54.492680 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:54.494162 (+ 1482us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:54.494615 (+ 453us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:54.494616 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:54.494616 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:54.494647 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:54.494811 (+ 164us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:54.494813 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:54.494813 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:54.494907 (+ 94us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:54.495070 (+ 163us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:54.495136 (+ 66us) server_negotiation.cc:300] Negotiation successful
0429 14:10:54.495223 (+ 87us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
Apr 29 14:10:54 dist-test-slave-8n77 krb5kdc[26266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471854, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.252@KRBTEST.COM for kudu/127.2.197.252@KRBTEST.COM
I20260429 14:10:54.499562 26506 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.492163 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:42465 (local address 127.0.0.1:52756)
0429 14:10:54.492234 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:54.492245 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:54.492292 (+ 47us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:54.492413 (+ 121us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:54.492413 (+ 0us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:54.492551 (+ 138us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:10:54.492716 (+ 165us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:54.492718 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.493548 (+ 830us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:54.493549 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:54.494081 (+ 532us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:54.494084 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.494159 (+ 75us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:54.494577 (+ 418us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:10:54.494582 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:10:54.495862 (+ 1280us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:10:54.497925 (+ 2063us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:54.497926 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:54.497926 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:54.498172 (+ 246us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:54.498449 (+ 277us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:54.498449 (+ 0us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:54.498450 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:54.498482 (+ 32us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:54.498883 (+ 401us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:10:54.498884 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:10:54.498936 (+ 52us) client_negotiation.cc:770] Sending connection context
0429 14:10:54.499014 (+ 78us) client_negotiation.cc:241] Negotiation successful
0429 14:10:54.499441 (+ 427us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14}
I20260429 14:10:54.499826 26508 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.492196 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:52756 (local address 127.2.197.252:42465)
0429 14:10:54.492234 (+ 38us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:54.492236 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:54.492277 (+ 41us) server_negotiation.cc:408] Connection header received
0429 14:10:54.492345 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:54.492346 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:54.492357 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:54.492411 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:54.492801 (+ 390us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.493443 (+ 642us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:54.494171 (+ 728us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.494416 (+ 245us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:54.495950 (+ 1534us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:54.495953 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:54.495953 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:54.495959 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:54.497824 (+ 1865us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:54.498266 (+ 442us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:54.498266 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:54.498267 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:54.498360 (+ 93us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:54.498568 (+ 208us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:54.498569 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:54.498570 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:54.498659 (+ 89us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:54.498729 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:54.499065 (+ 336us) server_negotiation.cc:300] Negotiation successful
0429 14:10:54.499551 (+ 486us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"mutex_wait_us":395,"server-negotiator.queue_time_us":19}
Apr 29 14:10:54 dist-test-slave-8n77 krb5kdc[26266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.197.253@KRBTEST.COM
I20260429 14:10:54.501557 26504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62dad35d86e44387be5e07fcb6f3a6c0: Bootstrap starting.
I20260429 14:10:54.502403 26504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62dad35d86e44387be5e07fcb6f3a6c0: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:54.502694 26504 log.cc:826] T 00000000000000000000000000000000 P 62dad35d86e44387be5e07fcb6f3a6c0: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:54.503491 26504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62dad35d86e44387be5e07fcb6f3a6c0: No bootstrap required, opened a new log
I20260429 14:10:54.504693 26432 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.496116 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34162 (local address 127.2.197.253:46827)
0429 14:10:54.496156 (+ 40us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:54.496157 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:54.496165 (+ 8us) server_negotiation.cc:408] Connection header received
0429 14:10:54.496236 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:54.496237 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:54.496246 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:54.496291 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:54.497276 (+ 985us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.498010 (+ 734us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:54.498960 (+ 950us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.499162 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:54.501967 (+ 2805us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:54.501972 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:54.501973 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:54.501981 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:54.503700 (+ 1719us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:54.504007 (+ 307us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:54.504008 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:54.504009 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:54.504034 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:54.504212 (+ 178us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:54.504212 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:54.504213 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:54.504327 (+ 114us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:54.504460 (+ 133us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:54.504507 (+ 47us) server_negotiation.cc:300] Negotiation successful
0429 14:10:54.504601 (+ 94us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260429 14:10:54.505175 26504 raft_consensus.cc:359] T 00000000000000000000000000000000 P 62dad35d86e44387be5e07fcb6f3a6c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "192a0a5f3eed499b9a52d956b6a2d109" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 35475 } } peers { permanent_uuid: "d7064b775fdd4fe5bc1eccd89d6764cb" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 46827 } } peers { permanent_uuid: "62dad35d86e44387be5e07fcb6f3a6c0" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42465 } }
I20260429 14:10:54.505445 26504 raft_consensus.cc:740] T 00000000000000000000000000000000 P 62dad35d86e44387be5e07fcb6f3a6c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62dad35d86e44387be5e07fcb6f3a6c0, State: Initialized, Role: FOLLOWER
I20260429 14:10:54.505563 26504 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62dad35d86e44387be5e07fcb6f3a6c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "192a0a5f3eed499b9a52d956b6a2d109" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 35475 } } peers { permanent_uuid: "d7064b775fdd4fe5bc1eccd89d6764cb" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 46827 } } peers { permanent_uuid: "62dad35d86e44387be5e07fcb6f3a6c0" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42465 } }
I20260429 14:10:54.505913 26504 sys_catalog.cc:565] T 00000000000000000000000000000000 P 62dad35d86e44387be5e07fcb6f3a6c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:10:54.505864 26510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62dad35d86e44387be5e07fcb6f3a6c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "192a0a5f3eed499b9a52d956b6a2d109" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 35475 } } peers { permanent_uuid: "d7064b775fdd4fe5bc1eccd89d6764cb" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 46827 } } peers { permanent_uuid: "62dad35d86e44387be5e07fcb6f3a6c0" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42465 } } }
I20260429 14:10:54.506078 26510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62dad35d86e44387be5e07fcb6f3a6c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260429 14:10:54.506368 2839 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
W20260429 14:10:54.507856 26521 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 62dad35d86e44387be5e07fcb6f3a6c0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260429 14:10:54.507906 26521 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260429 14:10:54.507365 26324 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "192a0a5f3eed499b9a52d956b6a2d109"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54396
I20260429 14:10:54.508704 26436 master_runner.cc:186] Error getting master registration for 127.2.197.254:35475: OK, instance_id { permanent_uuid: "192a0a5f3eed499b9a52d956b6a2d109" instance_seqno: 1777471854108626 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260429 14:10:54.509930 26436 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260429 14:10:54.539808 26427 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42465 } attempt: 3
Apr 29 14:10:54 dist-test-slave-8n77 krb5kdc[26266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471854, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.253@KRBTEST.COM for kudu/127.2.197.252@KRBTEST.COM
I20260429 14:10:54.548053 26508 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.540351 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:52770 (local address 127.2.197.252:42465)
0429 14:10:54.540410 (+ 59us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:54.540413 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:54.540425 (+ 12us) server_negotiation.cc:408] Connection header received
0429 14:10:54.540455 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:54.540457 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:54.540468 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:54.540552 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:54.540963 (+ 411us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.541686 (+ 723us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:54.542543 (+ 857us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.542720 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:54.545380 (+ 2660us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:54.545385 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:54.545385 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:54.545394 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:54.546896 (+ 1502us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:54.547311 (+ 415us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:54.547312 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:54.547312 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:54.547337 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:54.547521 (+ 184us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:54.547522 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:54.547522 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:54.547643 (+ 121us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:54.547707 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:54.547900 (+ 193us) server_negotiation.cc:300] Negotiation successful
0429 14:10:54.547977 (+ 77us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260429 14:10:54.548074 26430 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.540228 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:42465 (local address 127.0.0.1:52770)
0429 14:10:54.540298 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:54.540332 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:54.540384 (+ 52us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:54.540559 (+ 175us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:54.540560 (+ 1us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:54.540745 (+ 185us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:10:54.540884 (+ 139us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:54.540886 (+ 2us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.541783 (+ 897us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:54.541784 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:54.542448 (+ 664us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:54.542451 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.542537 (+ 86us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:54.543033 (+ 496us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:10:54.543038 (+ 5us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:10:54.545264 (+ 2226us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:10:54.546971 (+ 1707us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:54.546972 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:54.546972 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:54.547192 (+ 220us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:54.547411 (+ 219us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:54.547412 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:54.547412 (+ 0us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:54.547442 (+ 30us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:54.547717 (+ 275us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:10:54.547719 (+ 2us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:10:54.547773 (+ 54us) client_negotiation.cc:770] Sending connection context
0429 14:10:54.547860 (+ 87us) client_negotiation.cc:241] Negotiation successful
0429 14:10:54.547966 (+ 106us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260429 14:10:54.549670 26427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7064b775fdd4fe5bc1eccd89d6764cb: Bootstrap starting.
I20260429 14:10:54.550642 26427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7064b775fdd4fe5bc1eccd89d6764cb: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:54.550921 26427 log.cc:826] T 00000000000000000000000000000000 P d7064b775fdd4fe5bc1eccd89d6764cb: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:54.551833 26427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7064b775fdd4fe5bc1eccd89d6764cb: No bootstrap required, opened a new log
I20260429 14:10:54.552873 26350 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42465 } attempt: 3
I20260429 14:10:54.553525 26427 raft_consensus.cc:359] T 00000000000000000000000000000000 P d7064b775fdd4fe5bc1eccd89d6764cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "192a0a5f3eed499b9a52d956b6a2d109" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 35475 } } peers { permanent_uuid: "d7064b775fdd4fe5bc1eccd89d6764cb" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 46827 } } peers { permanent_uuid: "62dad35d86e44387be5e07fcb6f3a6c0" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42465 } }
I20260429 14:10:54.553648 26427 raft_consensus.cc:740] T 00000000000000000000000000000000 P d7064b775fdd4fe5bc1eccd89d6764cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7064b775fdd4fe5bc1eccd89d6764cb, State: Initialized, Role: FOLLOWER
I20260429 14:10:54.553705 26427 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7064b775fdd4fe5bc1eccd89d6764cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "192a0a5f3eed499b9a52d956b6a2d109" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 35475 } } peers { permanent_uuid: "d7064b775fdd4fe5bc1eccd89d6764cb" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 46827 } } peers { permanent_uuid: "62dad35d86e44387be5e07fcb6f3a6c0" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42465 } }
I20260429 14:10:54.553983 26427 sys_catalog.cc:565] T 00000000000000000000000000000000 P d7064b775fdd4fe5bc1eccd89d6764cb [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:10:54.554013 26524 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7064b775fdd4fe5bc1eccd89d6764cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "192a0a5f3eed499b9a52d956b6a2d109" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 35475 } } peers { permanent_uuid: "d7064b775fdd4fe5bc1eccd89d6764cb" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 46827 } } peers { permanent_uuid: "62dad35d86e44387be5e07fcb6f3a6c0" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42465 } } }
I20260429 14:10:54.554340 26524 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7064b775fdd4fe5bc1eccd89d6764cb [sys.catalog]: This master's current role is: FOLLOWER
W20260429 14:10:54.556804 26535 catalog_manager.cc:1568] T 00000000000000000000000000000000 P d7064b775fdd4fe5bc1eccd89d6764cb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260429 14:10:54.556856 26535 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Apr 29 14:10:54 dist-test-slave-8n77 krb5kdc[26266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777471854, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.197.254@KRBTEST.COM for kudu/127.2.197.252@KRBTEST.COM
I20260429 14:10:54.556674 26359 master_runner.cc:186] Error getting master registration for 127.2.197.254:35475: OK, instance_id { permanent_uuid: "192a0a5f3eed499b9a52d956b6a2d109" instance_seqno: 1777471854108626 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260429 14:10:54.558387 26359 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260429 14:10:54.561887 26508 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.553332 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:52780 (local address 127.2.197.252:42465)
0429 14:10:54.553369 (+ 37us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:54.553371 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:54.553380 (+ 9us) server_negotiation.cc:408] Connection header received
0429 14:10:54.553467 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:54.553468 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:54.553478 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:54.553528 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:54.554005 (+ 477us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.554710 (+ 705us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:54.555644 (+ 934us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.555823 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:54.558827 (+ 3004us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:54.558832 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:54.558832 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:54.558840 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:54.560558 (+ 1718us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:54.560875 (+ 317us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:54.560876 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:54.560876 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:54.560902 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:54.561056 (+ 154us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:54.561057 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:54.561058 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:54.561431 (+ 373us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:54.561517 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:54.561699 (+ 182us) server_negotiation.cc:300] Negotiation successful
0429 14:10:54.561789 (+ 90us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260429 14:10:54.561887 26352 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.553263 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.2.197.252:42465 (local address 127.0.0.1:52780)
0429 14:10:54.553323 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0429 14:10:54.553338 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 14:10:54.553383 (+ 45us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 14:10:54.553531 (+ 148us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 14:10:54.553533 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 14:10:54.553710 (+ 177us) client_negotiation.cc:190] Negotiated authn=SASL
0429 14:10:54.553908 (+ 198us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:54.553912 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.554806 (+ 894us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 14:10:54.554807 (+ 1us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 14:10:54.555373 (+ 566us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 14:10:54.555376 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.555451 (+ 75us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:54.555904 (+ 453us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 14:10:54.555910 (+ 6us) client_negotiation.cc:657] calling sasl_client_start()
0429 14:10:54.558754 (+ 2844us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 14:10:54.560630 (+ 1876us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:54.560631 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:54.560632 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:54.560788 (+ 156us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:54.560972 (+ 184us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 14:10:54.560973 (+ 1us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 14:10:54.560974 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 14:10:54.560996 (+ 22us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 14:10:54.561505 (+ 509us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 14:10:54.561506 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 14:10:54.561590 (+ 84us) client_negotiation.cc:770] Sending connection context
0429 14:10:54.561672 (+ 82us) client_negotiation.cc:241] Negotiation successful
0429 14:10:54.561790 (+ 118us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260429 14:10:54.563635 26350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 192a0a5f3eed499b9a52d956b6a2d109: Bootstrap starting.
I20260429 14:10:54.564317 26350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 192a0a5f3eed499b9a52d956b6a2d109: Neither blocks nor log segments found. Creating new log.
I20260429 14:10:54.564601 26350 log.cc:826] T 00000000000000000000000000000000 P 192a0a5f3eed499b9a52d956b6a2d109: Log is configured to *not* fsync() on all Append() calls
I20260429 14:10:54.565389 26350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 192a0a5f3eed499b9a52d956b6a2d109: No bootstrap required, opened a new log
I20260429 14:10:54.566491 26350 raft_consensus.cc:359] T 00000000000000000000000000000000 P 192a0a5f3eed499b9a52d956b6a2d109 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "192a0a5f3eed499b9a52d956b6a2d109" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 35475 } } peers { permanent_uuid: "d7064b775fdd4fe5bc1eccd89d6764cb" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 46827 } } peers { permanent_uuid: "62dad35d86e44387be5e07fcb6f3a6c0" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42465 } }
I20260429 14:10:54.566612 26350 raft_consensus.cc:740] T 00000000000000000000000000000000 P 192a0a5f3eed499b9a52d956b6a2d109 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 192a0a5f3eed499b9a52d956b6a2d109, State: Initialized, Role: FOLLOWER
I20260429 14:10:54.566682 26350 consensus_queue.cc:260] T 00000000000000000000000000000000 P 192a0a5f3eed499b9a52d956b6a2d109 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "192a0a5f3eed499b9a52d956b6a2d109" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 35475 } } peers { permanent_uuid: "d7064b775fdd4fe5bc1eccd89d6764cb" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 46827 } } peers { permanent_uuid: "62dad35d86e44387be5e07fcb6f3a6c0" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42465 } }
I20260429 14:10:54.566969 26350 sys_catalog.cc:565] T 00000000000000000000000000000000 P 192a0a5f3eed499b9a52d956b6a2d109 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 14:10:54.567034 26324 raft_consensus.cc:493] T 00000000000000000000000000000000 P 192a0a5f3eed499b9a52d956b6a2d109 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260429 14:10:54.567081 26324 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 192a0a5f3eed499b9a52d956b6a2d109 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:10:54.567976 26324 raft_consensus.cc:515] T 00000000000000000000000000000000 P 192a0a5f3eed499b9a52d956b6a2d109 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "192a0a5f3eed499b9a52d956b6a2d109" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 35475 } } peers { permanent_uuid: "d7064b775fdd4fe5bc1eccd89d6764cb" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 46827 } } peers { permanent_uuid: "62dad35d86e44387be5e07fcb6f3a6c0" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42465 } }
I20260429 14:10:54.568219 26324 leader_election.cc:290] T 00000000000000000000000000000000 P 192a0a5f3eed499b9a52d956b6a2d109 [CANDIDATE]: Term 1 election: Requested vote from peers d7064b775fdd4fe5bc1eccd89d6764cb (127.2.197.253:46827), 62dad35d86e44387be5e07fcb6f3a6c0 (127.2.197.252:42465)
I20260429 14:10:54.566967 26538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 192a0a5f3eed499b9a52d956b6a2d109 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "192a0a5f3eed499b9a52d956b6a2d109" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 35475 } } peers { permanent_uuid: "d7064b775fdd4fe5bc1eccd89d6764cb" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 46827 } } peers { permanent_uuid: "62dad35d86e44387be5e07fcb6f3a6c0" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42465 } } }
I20260429 14:10:54.568568 26538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 192a0a5f3eed499b9a52d956b6a2d109 [sys.catalog]: This master's current role is: FOLLOWER
I20260429 14:10:54.568682 26401 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "192a0a5f3eed499b9a52d956b6a2d109" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d7064b775fdd4fe5bc1eccd89d6764cb"
I20260429 14:10:54.568791 26401 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d7064b775fdd4fe5bc1eccd89d6764cb [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:10:54.569454 26401 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d7064b775fdd4fe5bc1eccd89d6764cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 192a0a5f3eed499b9a52d956b6a2d109 in term 1.
I20260429 14:10:54.570443 26478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "192a0a5f3eed499b9a52d956b6a2d109" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "62dad35d86e44387be5e07fcb6f3a6c0"
I20260429 14:10:54.570562 26478 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 62dad35d86e44387be5e07fcb6f3a6c0 [term 0 FOLLOWER]: Advancing to term 1
I20260429 14:10:54.571501 26478 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 62dad35d86e44387be5e07fcb6f3a6c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 192a0a5f3eed499b9a52d956b6a2d109 in term 1.
I20260429 14:10:54.571720 26299 leader_election.cc:304] T 00000000000000000000000000000000 P 192a0a5f3eed499b9a52d956b6a2d109 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 192a0a5f3eed499b9a52d956b6a2d109, 62dad35d86e44387be5e07fcb6f3a6c0; no voters:
I20260429 14:10:54.571890 26538 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 192a0a5f3eed499b9a52d956b6a2d109 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 14:10:54.572014 26538 raft_consensus.cc:697] T 00000000000000000000000000000000 P 192a0a5f3eed499b9a52d956b6a2d109 [term 1 LEADER]: Becoming Leader. State: Replica: 192a0a5f3eed499b9a52d956b6a2d109, State: Running, Role: LEADER
I20260429 14:10:54.572113 26538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 192a0a5f3eed499b9a52d956b6a2d109 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "192a0a5f3eed499b9a52d956b6a2d109" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 35475 } } peers { permanent_uuid: "d7064b775fdd4fe5bc1eccd89d6764cb" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 46827 } } peers { permanent_uuid: "62dad35d86e44387be5e07fcb6f3a6c0" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42465 } }
I20260429 14:10:54.572530 26538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 192a0a5f3eed499b9a52d956b6a2d109 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 192a0a5f3eed499b9a52d956b6a2d109. Latest consensus state: current_term: 1 leader_uuid: "192a0a5f3eed499b9a52d956b6a2d109" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "192a0a5f3eed499b9a52d956b6a2d109" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 35475 } } peers { permanent_uuid: "d7064b775fdd4fe5bc1eccd89d6764cb" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 46827 } } peers { permanent_uuid: "62dad35d86e44387be5e07fcb6f3a6c0" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42465 } } }
I20260429 14:10:54.572599 26538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 192a0a5f3eed499b9a52d956b6a2d109 [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:54.573369 26558 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 14:10:54.573846 26558 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 14:10:54.575492 26401 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d7064b775fdd4fe5bc1eccd89d6764cb [term 1 FOLLOWER]: Refusing update from remote peer 192a0a5f3eed499b9a52d956b6a2d109: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:10:54.575522 26478 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 62dad35d86e44387be5e07fcb6f3a6c0 [term 1 FOLLOWER]: Refusing update from remote peer 192a0a5f3eed499b9a52d956b6a2d109: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 14:10:54.575727 26538 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 192a0a5f3eed499b9a52d956b6a2d109 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7064b775fdd4fe5bc1eccd89d6764cb" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 46827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:10:54.575865 26538 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 192a0a5f3eed499b9a52d956b6a2d109 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62dad35d86e44387be5e07fcb6f3a6c0" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 14:10:54.576923 26524 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7064b775fdd4fe5bc1eccd89d6764cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 192a0a5f3eed499b9a52d956b6a2d109. Latest consensus state: current_term: 1 leader_uuid: "192a0a5f3eed499b9a52d956b6a2d109" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "192a0a5f3eed499b9a52d956b6a2d109" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 35475 } } peers { permanent_uuid: "d7064b775fdd4fe5bc1eccd89d6764cb" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 46827 } } peers { permanent_uuid: "62dad35d86e44387be5e07fcb6f3a6c0" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42465 } } }
I20260429 14:10:54.577042 26524 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7064b775fdd4fe5bc1eccd89d6764cb [sys.catalog]: This master's current role is: FOLLOWER
I20260429 14:10:54.578166 26510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62dad35d86e44387be5e07fcb6f3a6c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 192a0a5f3eed499b9a52d956b6a2d109. Latest consensus state: current_term: 1 leader_uuid: "192a0a5f3eed499b9a52d956b6a2d109" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "192a0a5f3eed499b9a52d956b6a2d109" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 35475 } } peers { permanent_uuid: "d7064b775fdd4fe5bc1eccd89d6764cb" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 46827 } } peers { permanent_uuid: "62dad35d86e44387be5e07fcb6f3a6c0" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42465 } } }
I20260429 14:10:54.578313 26510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62dad35d86e44387be5e07fcb6f3a6c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260429 14:10:54.579828 26538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 192a0a5f3eed499b9a52d956b6a2d109 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "192a0a5f3eed499b9a52d956b6a2d109" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "192a0a5f3eed499b9a52d956b6a2d109" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 35475 } } peers { permanent_uuid: "d7064b775fdd4fe5bc1eccd89d6764cb" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 46827 } } peers { permanent_uuid: "62dad35d86e44387be5e07fcb6f3a6c0" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42465 } } }
I20260429 14:10:54.579909 26538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 192a0a5f3eed499b9a52d956b6a2d109 [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:54.580300 26510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62dad35d86e44387be5e07fcb6f3a6c0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "192a0a5f3eed499b9a52d956b6a2d109" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "192a0a5f3eed499b9a52d956b6a2d109" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 35475 } } peers { permanent_uuid: "d7064b775fdd4fe5bc1eccd89d6764cb" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 46827 } } peers { permanent_uuid: "62dad35d86e44387be5e07fcb6f3a6c0" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42465 } } }
I20260429 14:10:54.580385 26510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62dad35d86e44387be5e07fcb6f3a6c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260429 14:10:54.580410 26561 mvcc.cc:204] Tried to move back new op lower bound from 7280524716338507776 to 7280524716328067072. Current Snapshot: MvccSnapshot[applied={T|T < 7280524716338507776}]
I20260429 14:10:54.580685 26564 mvcc.cc:204] Tried to move back new op lower bound from 7280524716338507776 to 7280524716328067072. Current Snapshot: MvccSnapshot[applied={T|T < 7280524716338507776}]
I20260429 14:10:54.580818 26538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 192a0a5f3eed499b9a52d956b6a2d109 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "192a0a5f3eed499b9a52d956b6a2d109" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "192a0a5f3eed499b9a52d956b6a2d109" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 35475 } } peers { permanent_uuid: "d7064b775fdd4fe5bc1eccd89d6764cb" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 46827 } } peers { permanent_uuid: "62dad35d86e44387be5e07fcb6f3a6c0" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42465 } } }
I20260429 14:10:54.580904 26538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 192a0a5f3eed499b9a52d956b6a2d109 [sys.catalog]: This master's current role is: LEADER
I20260429 14:10:54.581298 26524 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7064b775fdd4fe5bc1eccd89d6764cb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "192a0a5f3eed499b9a52d956b6a2d109" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "192a0a5f3eed499b9a52d956b6a2d109" member_type: VOTER last_known_addr { host: "127.2.197.254" port: 35475 } } peers { permanent_uuid: "d7064b775fdd4fe5bc1eccd89d6764cb" member_type: VOTER last_known_addr { host: "127.2.197.253" port: 46827 } } peers { permanent_uuid: "62dad35d86e44387be5e07fcb6f3a6c0" member_type: VOTER last_known_addr { host: "127.2.197.252" port: 42465 } } }
I20260429 14:10:54.581387 26524 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7064b775fdd4fe5bc1eccd89d6764cb [sys.catalog]: This master's current role is: FOLLOWER
I20260429 14:10:54.582726 26558 catalog_manager.cc:1357] Generated new cluster ID: 008c7b143ef749f19d080013b69f8dbf
I20260429 14:10:54.582783 26558 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 14:10:54.589769 26432 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.571769 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34166 (local address 127.2.197.253:46827)
0429 14:10:54.571823 (+ 54us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:54.571825 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:54.578470 (+ 6645us) server_negotiation.cc:408] Connection header received
0429 14:10:54.578540 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:54.578541 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:54.578554 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:54.578637 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:54.579657 (+ 1020us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.580314 (+ 657us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:54.582025 (+ 1711us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.582208 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:54.585509 (+ 3301us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:54.585514 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:54.585515 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:54.585523 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:54.587163 (+ 1640us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:54.588082 (+ 919us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:54.588084 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:54.588084 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:54.588111 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:54.589177 (+ 1066us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:54.589178 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:54.589179 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:54.589303 (+ 124us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:54.589381 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:54.589585 (+ 204us) server_negotiation.cc:300] Negotiation successful
0429 14:10:54.589670 (+ 85us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260429 14:10:54.594501 26508 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.570903 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:52792 (local address 127.2.197.252:42465)
0429 14:10:54.570947 (+ 44us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:54.570949 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:54.578637 (+ 7688us) server_negotiation.cc:408] Connection header received
0429 14:10:54.578729 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:54.578729 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:54.578740 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:54.578770 (+ 30us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:54.579762 (+ 992us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.580445 (+ 683us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:54.581213 (+ 768us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.581382 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:54.591209 (+ 9827us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:54.591214 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:54.591215 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:54.591223 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:54.593037 (+ 1814us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:54.593414 (+ 377us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:54.593415 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:54.593415 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:54.593464 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:54.593676 (+ 212us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:54.593677 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:54.593677 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:54.593809 (+ 132us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:54.593908 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:54.594318 (+ 410us) server_negotiation.cc:300] Negotiation successful
0429 14:10:54.594403 (+ 85us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260429 14:10:54.596868 26558 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 14:10:54.597584 26353 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:54.571119 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54398 (local address 127.2.197.254:35475)
0429 14:10:54.571168 (+ 49us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:54.571170 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:54.582453 (+ 11283us) server_negotiation.cc:408] Connection header received
0429 14:10:54.582559 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:54.582560 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:54.582571 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:54.582601 (+ 30us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:54.584159 (+ 1558us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.584810 (+ 651us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:54.587554 (+ 2744us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:54.587741 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:54.590759 (+ 3018us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 14:10:54.590763 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 14:10:54.590763 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 14:10:54.590771 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0429 14:10:54.594125 (+ 3354us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:54.596840 (+ 2715us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:54.596841 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:54.596842 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:54.596872 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 14:10:54.597048 (+ 176us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 14:10:54.597049 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 14:10:54.597049 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0429 14:10:54.597143 (+ 94us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 14:10:54.597217 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:54.597408 (+ 191us) server_negotiation.cc:300] Negotiation successful
0429 14:10:54.597506 (+ 98us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260429 14:10:54.597998 26558 catalog_manager.cc:1514] Loading token signing keys...
I20260429 14:10:54.607290 26558 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 192a0a5f3eed499b9a52d956b6a2d109: Generated new TSK 0
I20260429 14:10:54.607640 26558 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260429 14:10:54.631059 26353 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:10:54.629336 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54412 (local address 127.2.197.254:35475)
0429 14:10:54.629385 (+ 49us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:54.629388 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:54.630394 (+ 1006us) server_negotiation.cc:408] Connection header received
0429 14:10:54.630504 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:54.630505 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:54.630516 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:54.630548 (+ 32us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:54.630993 (+ 445us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54412: BlockingRecv error: recv got EOF from 127.0.0.1:54412 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260429 14:10:54.631789 26432 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:10:54.629783 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34170 (local address 127.2.197.253:46827)
0429 14:10:54.629821 (+ 38us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:54.629823 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:54.631306 (+ 1483us) server_negotiation.cc:408] Connection header received
0429 14:10:54.631403 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:54.631404 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:54.631415 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:54.631444 (+ 29us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:54.631736 (+ 292us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34170: BlockingRecv error: recv got EOF from 127.0.0.1:34170 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260429 14:10:54.632493 26508 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:10:54.630076 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:52808 (local address 127.2.197.252:42465)
0429 14:10:54.630114 (+ 38us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:54.630116 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:54.631975 (+ 1859us) server_negotiation.cc:408] Connection header received
0429 14:10:54.632096 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:54.632097 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:54.632107 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:54.632136 (+ 29us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:54.632441 (+ 305us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52808: BlockingRecv error: recv got EOF from 127.0.0.1:52808 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260429 14:10:54.634970 26432 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:10:54.634164 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34184 (local address 127.2.197.253:46827)
0429 14:10:54.634212 (+ 48us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:54.634214 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:54.634465 (+ 251us) server_negotiation.cc:408] Connection header received
0429 14:10:54.634581 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:54.634582 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:54.634595 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:54.634624 (+ 29us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:54.634922 (+ 298us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34184: BlockingRecv error: recv got EOF from 127.0.0.1:34184 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260429 14:10:54.638257 26432 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:10:54.637494 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34196 (local address 127.2.197.253:46827)
0429 14:10:54.637526 (+ 32us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:54.637528 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:54.637763 (+ 235us) server_negotiation.cc:408] Connection header received
0429 14:10:54.637859 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:54.637860 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:54.637869 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:54.637899 (+ 30us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:54.638212 (+ 313us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34196: BlockingRecv error: recv got EOF from 127.0.0.1:34196 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260429 14:10:54.642470 26432 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:10:54.641641 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34206 (local address 127.2.197.253:46827)
0429 14:10:54.641676 (+ 35us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:54.641679 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:54.641916 (+ 237us) server_negotiation.cc:408] Connection header received
0429 14:10:54.642012 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:54.642013 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:54.642023 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:54.642054 (+ 31us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:54.642422 (+ 368us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34206: BlockingRecv error: recv got EOF from 127.0.0.1:34206 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260429 14:10:54.648981 26432 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:10:54.647919 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34212 (local address 127.2.197.253:46827)
0429 14:10:54.647965 (+ 46us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:54.647967 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:54.648302 (+ 335us) server_negotiation.cc:408] Connection header received
0429 14:10:54.648468 (+ 166us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:54.648469 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:54.648482 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:54.648518 (+ 36us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:54.648921 (+ 403us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34212: BlockingRecv error: recv got EOF from 127.0.0.1:34212 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260429 14:10:54.659475 26432 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:10:54.658508 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34218 (local address 127.2.197.253:46827)
0429 14:10:54.658553 (+ 45us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:54.658556 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:54.658848 (+ 292us) server_negotiation.cc:408] Connection header received
0429 14:10:54.658958 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:54.658959 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:54.658971 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:54.659001 (+ 30us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:54.659389 (+ 388us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34218: BlockingRecv error: recv got EOF from 127.0.0.1:34218 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260429 14:10:54.677991 26432 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:10:54.677023 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34232 (local address 127.2.197.253:46827)
0429 14:10:54.677070 (+ 47us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:54.677073 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:54.677372 (+ 299us) server_negotiation.cc:408] Connection header received
0429 14:10:54.677491 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:54.677492 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:54.677504 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:54.677538 (+ 34us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:54.677928 (+ 390us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34232: BlockingRecv error: recv got EOF from 127.0.0.1:34232 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260429 14:10:54.712716 26432 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:10:54.711630 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34238 (local address 127.2.197.253:46827)
0429 14:10:54.711698 (+ 68us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:54.711700 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:54.712016 (+ 316us) server_negotiation.cc:408] Connection header received
0429 14:10:54.712158 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:54.712160 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:54.712174 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:54.712209 (+ 35us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:54.712653 (+ 444us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34238: BlockingRecv error: recv got EOF from 127.0.0.1:34238 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260429 14:10:54.779238 26432 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:10:54.778268 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34250 (local address 127.2.197.253:46827)
0429 14:10:54.778345 (+ 77us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:54.778348 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:54.778437 (+ 89us) server_negotiation.cc:408] Connection header received
0429 14:10:54.778501 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:54.778502 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:54.778516 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:54.778579 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:54.779173 (+ 594us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34250: BlockingRecv error: recv got EOF from 127.0.0.1:34250 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260429 14:10:54.909230 26432 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:10:54.908556 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34264 (local address 127.2.197.253:46827)
0429 14:10:54.908649 (+ 93us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:54.908651 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:54.908676 (+ 25us) server_negotiation.cc:408] Connection header received
0429 14:10:54.908754 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:54.908755 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:54.908766 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:54.908832 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:54.909144 (+ 312us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34264: BlockingRecv error: recv got EOF from 127.0.0.1:34264 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260429 14:10:55.167591 26432 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 14:10:55.166650 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34280 (local address 127.2.197.253:46827)
0429 14:10:55.166738 (+ 88us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:55.166740 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:55.166750 (+ 10us) server_negotiation.cc:408] Connection header received
0429 14:10:55.166778 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:55.166779 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:55.166789 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:55.166857 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0429 14:10:55.167500 (+ 643us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34280: BlockingRecv error: recv got EOF from 127.0.0.1:34280 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260429 14:10:55.508330 26521 catalog_manager.cc:1269] Loaded cluster ID: 008c7b143ef749f19d080013b69f8dbf
I20260429 14:10:55.508411 26521 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 62dad35d86e44387be5e07fcb6f3a6c0: loading cluster ID for follower catalog manager: success
I20260429 14:10:55.509337 26521 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 62dad35d86e44387be5e07fcb6f3a6c0: acquiring CA information for follower catalog manager: success
I20260429 14:10:55.509647 26521 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 62dad35d86e44387be5e07fcb6f3a6c0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260429 14:10:55.557302 26535 catalog_manager.cc:1269] Loaded cluster ID: 008c7b143ef749f19d080013b69f8dbf
I20260429 14:10:55.557412 26535 catalog_manager.cc:1562] T 00000000000000000000000000000000 P d7064b775fdd4fe5bc1eccd89d6764cb: loading cluster ID for follower catalog manager: success
I20260429 14:10:55.558357 26535 catalog_manager.cc:1584] T 00000000000000000000000000000000 P d7064b775fdd4fe5bc1eccd89d6764cb: acquiring CA information for follower catalog manager: success
I20260429 14:10:55.558634 26535 catalog_manager.cc:1612] T 00000000000000000000000000000000 P d7064b775fdd4fe5bc1eccd89d6764cb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260429 14:10:55.683403 26634 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:55.680825 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34290 (local address 127.2.197.253:46827)
0429 14:10:55.680958 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:55.680961 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:55.680972 (+ 11us) server_negotiation.cc:408] Connection header received
0429 14:10:55.681023 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:55.681024 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:55.681205 (+ 181us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:55.681355 (+ 150us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:10:55.681683 (+ 328us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:55.682190 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:55.682846 (+ 656us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:55.682999 (+ 153us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:55.683031 (+ 32us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:10:55.683123 (+ 92us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:10:55.683222 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:55.683280 (+ 58us) server_negotiation.cc:300] Negotiation successful
0429 14:10:55.683304 (+ 24us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260429 14:10:55.687954 26640 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:55.685551 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:52820 (local address 127.2.197.252:42465)
0429 14:10:55.685676 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:55.685678 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:55.685686 (+ 8us) server_negotiation.cc:408] Connection header received
0429 14:10:55.685728 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:55.685728 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:55.685753 (+ 25us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:55.685825 (+ 72us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:10:55.686078 (+ 253us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:55.686681 (+ 603us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:55.687276 (+ 595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:55.687455 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:55.687509 (+ 54us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:10:55.687599 (+ 90us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:10:55.687736 (+ 137us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:55.687808 (+ 72us) server_negotiation.cc:300] Negotiation successful
0429 14:10:55.687832 (+ 24us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":78,"threads_started":1}
I20260429 14:10:55.692373 26634 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:55.689963 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34300 (local address 127.2.197.253:46827)
0429 14:10:55.690011 (+ 48us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:55.690013 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:55.690023 (+ 10us) server_negotiation.cc:408] Connection header received
0429 14:10:55.690052 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:55.690053 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:55.690064 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:55.690147 (+ 83us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:10:55.690569 (+ 422us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:55.691063 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:55.691879 (+ 816us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:55.692038 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:55.692056 (+ 18us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:10:55.692080 (+ 24us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:10:55.692151 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:55.692251 (+ 100us) server_negotiation.cc:300] Negotiation successful
0429 14:10:55.692277 (+ 26us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260429 14:10:55.692862 26640 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:55.690129 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:52836 (local address 127.2.197.252:42465)
0429 14:10:55.690165 (+ 36us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:55.690167 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:55.690310 (+ 143us) server_negotiation.cc:408] Connection header received
0429 14:10:55.690360 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:55.690361 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:55.690373 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:55.690430 (+ 57us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:10:55.691036 (+ 606us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:55.691552 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:55.692380 (+ 828us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:55.692561 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:55.692584 (+ 23us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:10:55.692611 (+ 27us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:10:55.692659 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:55.692755 (+ 96us) server_negotiation.cc:300] Negotiation successful
0429 14:10:55.692784 (+ 29us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260429 14:10:55.692997 26647 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 14:10:55.689909 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54414 (local address 127.2.197.254:35475)
0429 14:10:55.690227 (+ 318us) server_negotiation.cc:207] Beginning negotiation
0429 14:10:55.690229 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0429 14:10:55.690240 (+ 11us) server_negotiation.cc:408] Connection header received
0429 14:10:55.690324 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 14:10:55.690325 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 14:10:55.690356 (+ 31us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 14:10:55.690430 (+ 74us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 14:10:55.690733 (+ 303us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:55.691627 (+ 894us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 14:10:55.692380 (+ 753us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 14:10:55.692552 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 14:10:55.692603 (+ 51us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 14:10:55.692705 (+ 102us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 14:10:55.692763 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
0429 14:10:55.692866 (+ 103us) server_negotiation.cc:300] Negotiation successful
0429 14:10:55.692892 (+ 26us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":53,"threads_started":1}
I20260429 14:10:55.695482 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 26282
I20260429 14:10:55.701192 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 26359
I20260429 14:10:55.707666 2839 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskhpfkRA/build/release/bin/kudu with pid 26436
2026-04-29T14:10:55Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3928 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (7876 ms total)
[----------] Global test environment tear-down
[==========] 57 tests from 4 test suites ran. (442217 ms total)
[ PASSED ] 56 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestEncryptionWithKMSIntegration
1 FAILED TEST